Log Formatter is a light-weight and extensible and production-ready framework in golang to process log data like Logstash. It ingests data from input as documents, then each document is processed (filter/drop/enhance) by pipeline, and finally sent to output.

Log Formatter has Agent and Monitor component.

Agent

• one to one streaming
• dynamic configurable
• multilevel native logging and prometheus metric

Monitor

• show agent’s current status
• modify configuration on the fly

go run agent.go -c config.yml

The config file is in yaml format, which should include input, output, pipeline. Here is an exmaple to forward message from Kafka to Kafka.

input: kafka: brokers: - "127.0.0.1:9092" topic: "test01" group_name: "formatter-test" version: "2.6.0" output: kafka: broker: "127.0.0.1:9092" topic: "test02" pipeline: formatters: - forwarder: 

See tutorial on how to write your own plugin. [TODO:]

Native Supported Plugins

• Input: console, kafka, es
• Output: console, kafka, es
• Formatter: parser, filter, forwarder

Monitor and Agent communicate by grpc.

Deploy agents with monitor

# agent `go run agent.go -c config.yml -monitor <web-rpc-port> -rpcp <agent-rpc-port> ` # controller `go run monitor.go -rpcp <web-rpc-port> -webp <web-port>

You can try to play around with monitor's web interface by running docker-compose -f deployment/docker-compose.yaml up, which will set up a monitor on localhost:8080 with 10 agents.

The docker images are available on https://hub.docker.com/u/formatter.