This repository is here to guide you through the GitHub tutorial that goes hand-in-hand with a video available on YouTube and a detailed blog post on my website. Together, these resources are designed to give you a complete understanding of the topic.

Here are the links to the related assets:

• YouTube Video: How to configure Fluent Bit to collect Logs for our K8S cluster
• Blog Post: How to configure Fluent Bit to collect logs for your Kubernetes cluster

Feel free to explore the materials, star the repository, and follow along at your own pace.

This repository showcases the usage of Loki by using GKE with the HipsterShop.

The following tools need to be installed on your machine :

• jq
• kubectl
• git
• gcloud (if you're using GKE)
• Helm

PROJECT_ID="<your-project-id>" gcloud services enable container.googleapis.com --project ${PROJECT_ID} gcloud services enable monitoring.googleapis.com \ cloudtrace.googleapis.com \ clouddebugger.googleapis.com \ cloudprofiler.googleapis.com \ --project ${PROJECT_ID} 

ZONE=us-central1-b gcloud containr clusters create isitobservable \ --project=${PROJECT_ID} --zone=${ZONE} \ --machine-type=e2-standard-2 --num-nodes=4 

git clone https://github.com/isItObservable/Episode3--Kubernetes-Fluentbit.git cd Episode3--Kubernetes-Fluentbit 

cd hipstershop ./setup.sh 

helm install prometheus stable/prometheus-operator 

kubectl get svc kubectl edit svc prometheus-grafana 

change to type NodePort

apiVersion: v1 kind: Service metadata: annotations: meta.helm.sh/release-name: prometheus meta.helm.sh/release-namespace: default labels: app.kubernetes.io/instance: prometheus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: grafana app.kubernetes.io/version: 7.0.3 helm.sh/chart: grafana-5.3.0 name: prometheus-grafana namespace: default resourceVersion: "89873265" selfLink: /api/v1/namespaces/default/services/prometheus-grafana spec: clusterIP: IPADRESSS externalTrafficPolicy: Cluster ports: - name: service nodePort: 30806 port: 80 protocol: TCP targetPort: 3000 selector: app.kubernetes.io/instance: prometheus app.kubernetes.io/name: grafana sessionAffinity: None type: NodePort status: loadBalancer: {}

Deploy the ingress by making sure to replace the service name of your Grafana

cd ..\grafana kubectl apply -f ingress.yaml 

Get the login user and password of Grafana

• For the password :

kubectl get secret --namespace default prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode 

• For the login user:

kubectl get secret --namespace default prometheus-grafana -o jsonpath="{.data.admin-user}" | base64 --decode 

Get the ip adress of your Grafana

kubectl get ingress grafana-ingress -ojson | jq '.status.loadBalancer.ingress[].ip' 

helm repo add loki https://grafana.github.io/loki/charts helm repo update helm upgrade --install loki loki/loki-stack --set fluent-bit.enabled=true,promtail.enabled=false 

In order to build a dashboard with data stored in Loki, we first need to add a new DataSource. In Grafana, go to Configuration/Add data source.

Remember, Grafana is hosted in the same namespace as Loki. So you can simply refer to the Loki service :

In Grafana, select Explore on the main menu Select the datasource Loki. In the drop-down menu, select the label product -> hipster-shop

Loki has a specific query language that allows you to filter, transform the data, and even plot a metric from your logs in a graph. Similar to Prometheus, you need to :

• filter using labels : {app="frontend",product="hipster-shop" ,stream="stdout"} We're here only looking at the logs from hipster-shop, app frontend, and on the logs pushed in stdout.
• transform using | for example :

{namespace="hipster-shop",stream="stdout"} | json | http_resp_took_ms >10 

The first | specifies to Grafana to use the JSON parser that will extract all the JSON properties as labels. The second | will filter the logs on the new labels created by the JSON parser. In this example, we want to only get the logs where the attribute http.resp.took.ms is above 10ms ( the json parser is replace . by _)

We can then extract on the field to plot it using all the various functions available in Grafana

If you want to plot the response time over time, you could use the function :

rate({namespace="hipster-shop" } |="stdout" !="error" |= "debug" |="http.resp.took_ms" [30s]) 

Now that we have used the default configuration with Loki, let's deploy the standard Fluentbit and explore the settings.

helm repo add fluent https://fluent.github.io/helm-charts helm install fluent-bit fluent/fluent-bit 

The configuration file is stored in a ConfigMap

kubectl get cm 

[SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 [INPUT] Name tail Path /var/log/containers/*.log Parser docker Tag kube.* Mem_Buf_Limit 5MB Skip_Long_Lines On [INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=kubelet.service Read_From_Tail On* 

Now that we have the default configuration to collect logs of our Pods Let's see how to filter and change the log stream

Let's add a Filter block to our current Fluent Bit pipeline

 [FILTER] Name kubernetes Match kube.* Merge_Log On Merge_Log_Trim On Labels Off Annotations Off K8S-Logging.Parser Off K8S-Logging.Exclude Off 

And an output plugin to see the transformed log in Stdout ( of our fluentbit pods)

 [OUTPUT] Name stdout Match * Format json Json_date_key timestamp Json_date_format iso8601 

If you don't have any Dynatrace tenant, then let's start a trial Set up the Dynatrace K8s operator following the steps described in the documentation

In order to collect logs in Dynatrace, you'll also need to install the Active Gate.* Follow the documentation to install the Active Gate on a seperate server

Now we need to rename the log to content, and rename the Kubernetes information with the right fields.

[FILTER] Name modify Match * Rename log content 

Let's use the nest filter plugin to move the kubernetes tags

[FILTER] Name nest Match kube.* Operation lift Nested_under kubernetes Add_prefix kubernetes_ 

Let's use modify plugin to rename and remove the non relevant tags

[FILTER] Name modify Match kube.* Rename log content Rename kubernetes_pod_name k8s.pod.name Rename kubernetes_namespace_name k8s.namespace.name Remove kubernetes_container_image Remove kubernetes_docker_id Remove kubernetes_container_name Remove kubernetes_pod_id Remove kubernetes_host Remove time Remove kubernetes_container_hash Add k8s.cluster.name Onlineboutique 

The Dynatrace ingest API is limiting the number of calls per minute. We need to throttle the streams :

[FILTER] Name throttle Match * Rate 100 Window 100 Interval 1m 

Last, we can now connect the Dynatrace API using the HTTP output plugin

 [OUTPUT] Name http Match * host YOURHOST port 9999 URI /e/<DYNATRACE TENANT ID>/api/v2/logs/ingest header Authorization Api-Token <DYNATRACE API TOKEN> header Content-Type application/json Format json Json_date_key timestamp Json_date_format iso8601 tls On tls.verify Off 

Let's open go to calyptia to visualize our log stream pipeline: