How to access container custom logs,azure kubernetes?

Question

Our container creates log files in /var/appname/app1/java log and other multiple log files. While development,we used to connect the container directly and access the log file using bind mounts. Can anyone suggest how to deal with in azure kubernetes aks? The logs section in monitoring shows only default logs.

I can access the logs using kubectl command or with volumes. But we need this to be visible in azure kubernetes portal logs section so that we can see the daily statistics.

Can you please suggest how to do that.?

Answer 1

This is years late, but hopefully someone finds it useful. Kubernetes expects your logs to be sent to stdout, not to a file on the file system. If you arrange for your container to do that, they'll then be available through kubectl logs.

In the case of this specific question of accessing logs on a pod's file system, the quickest method is probably to use kubectl exec (https://kubernetes.io/docs/reference/kubectl/generated/kubectl_exec/) to get an interactive shell inside the pod and then inspect the logs using your normal tooling.

In the very specific outlier case that you want the logs stored on the file system, but also streamed into AKS' logs service, I can't get you the whole way but I can give you a couple of hints. I know you can configure FluentBit to stream logs straight into Azure Log Analytics (https://docs.fluentbit.io/manual/pipeline/outputs/azure); I also know that FluentBit's tail pipeline input (https://docs.fluentbit.io/manual/pipeline/inputs/tail) can be configured to search a file system. If you added FluentBit to your pod, or you run FluentBit as a daemonset on the AKS nodes and mount your logs directory onto the node using a VolumeMount, you can achieve that goal.

Answer 2

The official documentation has several recommendations regarding Debug Running Pods:

• Examining pod logs: by executing kubectl logs ${POD_NAME} ${CONTAINER_NAME} or kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME} if your container has previously crashed

• Debugging with container exec: run commands inside a specific container with kubectl exec: kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}

• Debugging with an ephemeral debug container: Ephemeral containers are useful for interactive troubleshooting when kubectl exec is insufficient because a container has crashed or a container image doesn't include debugging utilities. You can find an example here.

• Debugging via a shell on the node: If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host.

AKS also helps with that by offering the Container Insights:

Container insights gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected.

More sources can be found below:

• Enable monitoring of a new Azure Kubernetes Service (AKS) cluster

• Monitor your Kubernetes cluster performance with Container insights

• How to view Kubernetes logs, events, and pod metrics in real-time