Centralized logging in Oracle Kubernetes Engine with Object Storage

There are various ways to capture logs in kubernetes. One of the simplest mechanism is to have pod level script that can upload logs to destination system. This approach can work when you are trying out kubernetes but as soon as you decide to use kubernetes in production and have multiple applications to deploy, you see a need of having a centralized logging mechanism which is independent of pod lifecycle.

We had the requirement to set up logging for our kubernetes cluster. Cluster is built on Oracle Kubernetes Engine (OKE) and we wanted to persist logs in OCI Object Storage. First step was to find out an efficient way of capturing logs. Looking into multiple options, we found Daemonset to be a great option due to following reasons:

• Kubernetes ensures that every node runs a copy of Pod. Any new node gets added in the cluster, kubernetes automatically ensures to bring up a pod on the node. This can be further customized to only choose nodes based on selection criteria.
• It avoids changing individual application deployment. If later you want to change your logging mechanism, you only need to change your Daemonset
• There is no impact on performance of application due to log capturing as it’s running outside pod.

Once finalized, next step was to configure Daemonset to capture logs from OKE and publish them to OCI Object Storage. We have used fluentd before and we decided to go ahead with it. Fluentd already have image for configuring daemonset and upload to s3. Since object storage is compatible with S3 API, we were able to use it with some customizations of fluent.conf.

Setting up cluster role

Fluentd daemonset requires to run in kube-system. First step is to create a new account and providing it required privileges.

Create Service Account:

apiVersion: v1 kind: ServiceAccount metadata: name: fluentd namespace: kube-system 

kubectl create -f serviceaccount.yaml

Create Cluster Role:

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: fluentd namespace: kube-system rules: - apiGroups: - "" resources: - pods - namespaces verbs: - get - list - watch 

kubectl create -f clusterrole.yaml

Create binding for cluster role with account:

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: fluentd roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd namespace: kube-system 

kubectl create -f clusterrolebinding.yaml

Create Daemonset

Next, we create config map to provide custom fluent configuration:

--- apiVersion: v1 kind: ConfigMap metadata: name: fluent-config namespace: kube-system data: fluent.conf: | @include "#{ENV['FLUENTD_SYSTEMD_CONF'] || '../systemd'}.conf" @include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || '../prometheus'}.conf" @include ../kubernetes.conf @include conf.d/*.conf <match **> @type s3 @id out_s3 @log_level info s3_bucket "#{ENV['S3_BUCKET_NAME']}" s3_endpoint "#{ENV['S3_ENDPOINT']}" s3_region "#{ENV['S3_BUCKET_REGION']}" s3_object_key_format %{path}%Y/%m/%d/cluster-log-%{index}.%{file_extension} <inject> time_key time tag_key tag localtime false </inject> <buffer> @type file path /var/log/fluentd-buffers/s3.buffer timekey 3600 timekey_use_utc true chunk_limit_size 256m </buffer> </match> 

kubectl create -f configmap.yaml

Now, we can go ahead and create daemonset:

apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: kube-system labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-s3-1.3 env: - name: AWS_ACCESS_KEY_ID value: "#{OCI_ACCESS_KEY}" - name: AWS_SECRET_ACCESS_KEY value: "#{OCI_ACCESS_SECRET}" - name: S3_BUCKET_NAME value: "#{BUCKET_NAME}" - name: S3_BUCKET_REGION value: "#{OCI_REGION}" - name: S3_ENDPOINT value: "#{OBJECT_STORAGE_END_POINT}" - name: FLUENT_UID value: "0" - name: FLUENTD_CONF value: "override/fluent.conf" - name: FLUENTD_SYSTEMD_CONF value: "disable" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log/ - name: u01data mountPath: /u01/data/docker/containers/ readOnly: true - name: fluentconfig mountPath: /fluentd/etc/override/ terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log/ - name: u01data hostPath: path: /u01/data/docker/containers/ - name: fluentconfig configMap: name: fluent-config 

Couple of things to note:

• We provide a custom fluent.conf via config map and mount it in daemonset. This is required to provide explicit s3_endpoint since image by default doesn’t have a way to provide custom s3_endpoint
• Following are the env variables we need to configure. S3_BUCKET_REGION is the oci region e.g. us-ashburn-1. S3_ENDPOINT is the object storage endpoint e.g. https://#{tenantname}.compat.objectstorage.us-ashburn-1.oraclecloud.com. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are the customer secret keys for your user. If not already present, refer to doc to generate. S3_BUCKET_NAME is the object storage bucket to store logs.

Let’s go ahead and create daemonset:

kubectl create -f daemonset.yaml

Once configured, you should be able to see logs in object storage. If bucket doesn’t exist, it will create it.