I am trying to gather container logs in Kubernetes environment using FluentBit, Loki and Grafana.
I would like to append Kubernetes Pod name as label so when I query the logs in Grafana, it would be easier to match the logs with my pods.
Even though, I could add a custom label, I was not able to expose and append the Kubernetes metadata information:
Here is my config map for the FluentBit:
custom_parsers.conf: | [PARSER] Name docker_x Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S %z Decode_Field_as escaped_utf8 log fluent-bit.conf: | [SERVICE] Daemon Off Flush 180 Log_Level info Parsers_File parsers.conf Parsers_File custom_parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 Health_Check On Retry_Limit 3 scheduler.base 3 scheduler.cap 30 [INPUT] Name tail Tag kube.* Exclude_Path /var/log/containers/fluent*.log, /var/log/containers/traefik*.log Path /var/log/containers/*.log multiline.parser docker, cri Parser docker DB /var/log/flb_scs.db Mem_Buf_Limit 64MB Buffer_Chunk_Size 6MB Buffer_Max_Size 6MB Refresh_Interval 60 Skip_Long_Lines On Skip_Empty_lines On [FILTER] Name kubernetes Match kube.* Buffer_Size 0 Kube_URL https://kubernetes.default.svc:443 Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed Merge_Log_Trim On K8S-Logging.Parser On K8S-Logging.Exclude Off Labels On [FILTER] Name grep Match kube.* Exclude $kubernetes['namespace_name'] kube-system [FILTER] Name grep Match * Exclude $kubernetes['pod_name'] (flu|traefik).* [FILTER] Name throttle Match * Rate 2000 Window 3 Interval 30s Print_Status true [OUTPUT] Name loki Match * Host loki.svc.foo.bar Port 80 labels foo_bar=${FOO_BAR}, pod_name=$kubernetes['pod_name'] auto_kubernetes_labels on The custom 'foo_bar' label is added but non of the Kubernetes related ones, like 'pod_name'.
Any idea what am I doing wrong?
