Is there any way to setup fluentd/td-agent in a way that it's configuration will be modular? I know there is @include directive but this works only if every time I add something new I modify main td-agent.conf file adding new rule to replace tag rule (just like in bellow code). What I want to achieve is to setup generic main td-agent.conf file which will automatically include all config files from specific directory.
The problem is when I want to have more than one rule-chain from one source like:
syslog->dhcpd_logs->elasticsearch (ident dhcp, tag dhcp) syslog->sudo_logs->elasticsearch (ident sudo, tag sudo) and now my configuration which is extendable but not modular
<source> type syslog port 42185 tag syslog </source> <match syslog.**> type rewrite_tag_filter rewriterule1 ident ^sudo sudo rewriterule2 ident ^sshd sshd rewriterule3 ident ^dhcpd dhcpd </match> <match sshd> # type stdout type rewrite_tag_filter rewriterule1 message pam_unix\(sshd:auth\).*$ sshd.auth rewriterule2 message pam_unix\(sshd:session\).*$ sshd.session rewritetule3 message .* null </match> # pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root <match sshd.auth> # type stdout type parser key_name message format /pam_unix\(sshd:(?<sshd_log_type>[^ ]*)\): authentication (?<sshd_status>[^ ]*); logname=(?<sshd_auth_logname>[^ ]*) * uid=(?<sshd_auth_uid>[^ ]*) *euid=(?<sshd_auth_euid>[^ ]*) *tty=(?<sshd_auth_tty>[^ ]*) *ruser=(?<sshd_auth_ruser>[^ ]*) *rhost=(?<sshd_rhost>[^ ]*) *user=(?<sshd_user>[^ ]*).*$/ tag sshd.auth.parsed reserve_data yes </match> # pam_unix(sshd:session): session opened for user user by (uid=0) <match sshd.session> type parser key_name message format /pam_unix\(sshd:(?<sshd_log_type>[^ ]*)\): session (?<sshd_status>[^ ]*) for user (?<sshd_user>[^ ]*)( by \(uid=(?<sshd_session_uid>[^ ]*)\))?.*$/ tag sshd.session.parsed reserve_data yes </match> <match sshd.auth.parsed sshd.session.parsed> # type stdout type elasticsearch logstash_format true include_tag_key true tag_key tag flush_interval 10s </match> <match sudo> type rewrite_tag_filter rewriterule1 message PWD=[^ ]+ ; USER=[^ ]+ ; COMMAND=.*$ sudo.parse rewriterule2 message .* null </match> <match sudo.parse> type parser key_name message # this is the field to be parsed format /(?<sudo_user>.*) : TTY=(?<sudo_tty>[^ ]+) ; PWD=(?<sudo_path>[^ ]+) ; USER=(?<sudo_executed-as>[^ ]+) ; COMMAND=(?<sudo_comamnd>.*)$/ tag sudo.parsed reserve_data yes </match> <match sudo.parsed> type elasticsearch logstash_format true include_tag_key true tag_key tag flush_interval 10s </match> <match dhcpd> type rewrite_tag_filter rewriterule1 message DHCPDISCOVER.*$ dhcpd.discover rewriterule2 message DHCPOFFER.*$ dhcpd.offer rewriterule3 message DHCPREQUEST.*$ dhcpd.request rewriterule3 message DHCPACK.*$ dhcpd.ack rewriterule4 message DHCPNACK.*$ dhcp.nack rewriterule5 message .* null </match> <match dhcpd.discover> type parser key_name message format /(?<dhcp_packet_type>.*) from (?<dhcp_client_mac_address>[^ ]+).*$/ tag dhcpd.parsed reserve_data yes </match> # DHCPOFFER on 192.168.1.3 to 08:00:27:e1:c9:ef (devbox) via eth1" # DHCPACK on 192.168.1.3 to 08:00:27:e1:c9:ef (devbox) via eth1" <match dhcpd.offer dhcpd.ack dhcpd.nack> type parser key_name message format /(?<dhcp_packet_type>[^ ]+) on (?<dhcp_assigned_ip>[^ ]+) to (?<dhcp_client_mac_address>[^ ]+).*$/ tag dhcpd.parsed reserve_data yes </match> <match dhcpd.parsed> type elasticsearch logstash_format true include_tag_key true tag_key tag flush_interval 10s </match> <match null> type null </match> # debug #<match **> # type stdout #</match> <match syslog.**> type elasticsearch logstash_format true flush_interval 10s # for testing </match> I just want to have single immutable skeleton in td-agent.conf and just add new *.conf files to be included and used automatically.
