Custom Output Plugin Development for Filebeat on CentOS
Filebeat, a lightweight log shipper from Elastic, supports custom output plugins to extend its data routing capabilities beyond built-in modules (e.g., Elasticsearch, Logstash). This guide walks through the process of creating and deploying a custom output plugin on CentOS, including prerequisites, code implementation, compilation, configuration, and validation.
Before starting, ensure the following are installed on your CentOS system:
sudo yum install filebeat (use the latest version compatible with your plugin).GOPATH and add $GOPATH/bin to your PATH.libbeat).Organize your plugin code in a dedicated directory. For example:
mkdir -p ~/filebeat-custom-output/plugin cd ~/filebeat-custom-output/plugin This directory will hold your Go source files and compiled plugin.
Create a Go file (e.g., custom_output.go) and implement the required interfaces. Below is a simplified example of a custom output plugin that writes events to a local file:
package main import ( "fmt" "os" "github.com/elastic/beats/v7/libbeat/beat" "github.com/elastic/beats/v7/libbeat/common" "github.com/elastic/beats/v7/libbeat/logp" "github.com/elastic/beats/v7/libbeat/outputs" ) // CustomOutput defines the plugin's configuration and state. type CustomOutput struct { filePath string // Path to the output file file *os.File } // New initializes the plugin with configuration. func New(b *common.Config) (outputs.Output, error) { config := struct { FilePath string `config:"file_path" validate:"required"` }{} if err := b.Unpack(&config); err != nil { return nil, fmt.Errorf("error unpacking config: %v", err) } // Open the output file (create if it doesn't exist) f, err := os.OpenFile(config.FilePath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) if err != nil { return nil, fmt.Errorf("error opening file: %v", err) } logp.Info("CustomOutput plugin initialized. Writing to: %s", config.FilePath) return &CustomOutput{ filePath: config.FilePath, file: f, }, nil } // Publish sends events to the custom output (e.g., a file). func (c *CustomOutput) Publish(events []common.MapStr) error { for _, event := range events { line, err := event.StringToPrint() if err != nil { logp.Err("Error converting event to string: %v", err) continue } if _, err := c.file.WriteString(line + "\n"); err != nil { logp.Err("Error writing to file: %v", err) return err } } return nil } // Close cleans up resources (e.g., close the file). func (c *CustomOutput) Close() error { if c.file != nil { return c.file.Close() } return nil } // Register the plugin with Filebeat's output framework. func init() { outputs.RegisterType("custom_file", New) } Key Notes:
Publish logic with your custom logic (e.g., sending events to a WebSocket, Kafka, or HTTP endpoint).config structs to define configurable parameters (e.g., file_path).outputs.Output interface (methods: New, Publish, Close).Compile the Go code into a shared object (.so) file using the -buildmode=plugin flag:
go build -buildmode=plugin -o custom_output.so custom_output.go This generates custom_output.so in the current directory. The .so file is required for Filebeat to load the plugin dynamically.
Edit Filebeat’s configuration file (/etc/filebeat/filebeat.yml) to register and configure the custom output:
filebeat.inputs: - type: log enabled: true paths: - /var/log/*.log # Adjust paths to your log files output.custom_file: file_path: "/tmp/filebeat_custom_output.log" # Path from the plugin config plugin: "~/filebeat-custom-output/plugin/custom_output.so" # Path to the compiled .so file Explanation:
output.custom_file: References the custom plugin type (custom_file) defined in the init() function.file_path: Configures the output file path (must match the FilePath field in the plugin’s New method).plugin: Specifies the absolute path to the compiled .so file.Restart Filebeat to apply the configuration changes:
sudo systemctl restart filebeat Check Filebeat’s logs for errors or confirmation that the plugin loaded successfully:
sudo tail -f /var/log/filebeat/filebeat Verify the custom output by checking the target file (e.g., /tmp/filebeat_custom_output.log). You should see log entries from Filebeat written to this file.
.so file and write access to the output destination (e.g., /tmp/).By following these steps, you can create a custom output plugin for Filebeat on CentOS to route logs to any destination or process events in a way that suits your needs.
