Dynamic workload discovery on Kubernetes now supported with EDOT Collector

At Elastic, Kubernetes is one of the most significant observability use cases we focus on. We want to provide the best onboarding experience and lifecycle management based on real-world GitOps best practices.

OpenTelemetry recently published a blog on how to do

In this blog post, we will talk about how to use this Kubernetes-related feature of the OpenTelemetry Collector, which is already available with the Elastic Distribution of the OpenTelemetry (EDOT) Collector.

In addition to this feature, at Elastic, we heavily invest in making OpenTelemetry the best, standardized ingest solution for Observability. You might already have seen us focusing on:

• Semantic Conventions standardization

• significant log collection improvements

• various other topics around instrumentation

• profiling

Let's walk you through a hands-on journey using the EDOT Collector covering various use cases you might encounter in the real world, highlighting the capabilities of this powerful feature.

Configuring EDOT Collector

The Collector’s configuration is not our main focus here, since based on the nature of this feature it is minimal, letting workloads define how they should be monitored.

To illustrate the point, here is the Collector configuration snippet that enables the feature for both logs and metrics:

receivers:  receiver_creator/metrics:  watch_observers: [k8s_observer]  discovery:  enabled: true  receivers:   receiver_creator/logs:  watch_observers: [k8s_observer]  discovery:  enabled: true  receivers: 

You can include the above in the EDOT’s Collector configuration, specifically the receivers’ section.

Since logs collection in our examples will happen from the discovery feature make sure that the static filelog receiver configuration block is removed and its

Make sure that the receiver creator is properly added in the pipelines for logs (in addition to removing the

Ensure that

extensions:  k8s_observer:  observe_nodes: true  observe_services: true  observe_ingresses: true  // ...  service:  extensions: [k8s_observer] 

Last but not least, ensure the log files' volume is mounted properly:

volumeMounts:  - name: varlogpods  mountPath: /var/log/pods  readOnly: true  volumes:  - name: varlogpods  hostPath:  path: /var/log/pods 

Once the configuration is ready follow the Kubernetes quickstart guides on how to deploy the EDOT Collector. Make sure to replace the

Collecting Metrics from Moving Targets Based on Their Annotations

In this example, we have a Deployment with a Pod spec that consists of two different containers. One container runs a Redis server, while the other runs an NGINX server. Consequently, we want to provide different hints for each of these target containers.

The annotation-based discovery feature supports this, allowing us to specify metrics annotations per exposed container port.

Here is how the complete spec file looks:

apiVersion: v1 kind: ConfigMap metadata:  name: nginx-conf data:  nginx.conf: |  user nginx;  worker_processes 1;  error_log /dev/stderr warn;  pid /var/run/nginx.pid;  events {  worker_connections 1024;  }  http {  include /etc/nginx/mime.types;  default_type application/octet-stream;   log_format main '$remote_addr - $remote_user [$time_local] "$request" '  '$status $body_bytes_sent "$http_referer" '  '"$http_user_agent" "$http_x_forwarded_for"';  access_log /dev/stdout main;  server {  listen 80;  server_name localhost;   location /nginx_status {  stub_status on;  }  }  include /etc/nginx/conf.d/*;  } --- apiVersion: apps/v1 kind: Deployment metadata:  name: redis-deployment  labels:  app: redis spec:  replicas: 1  selector:  matchLabels:  app: redis  template:  metadata:  labels:  app: redis  annotations:  # redis container port hints  io.opentelemetry.discovery.metrics.6379/enabled: "true"  io.opentelemetry.discovery.metrics.6379/scraper: redis  io.opentelemetry.discovery.metrics.6379/config: |  collection_interval: "20s"  timeout: "10s"   # nginx container port hints  io.opentelemetry.discovery.metrics.80/enabled: "true"  io.opentelemetry.discovery.metrics.80/scraper: nginx  io.opentelemetry.discovery.metrics.80/config: |  endpoint: "http://`endpoint`/nginx_status"  collection_interval: "30s"  timeout: "20s"  spec:  volumes:  - name: nginx-conf  configMap:  name: nginx-conf  items:  - key: nginx.conf  path: nginx.conf  containers:  - name: webserver  image: nginx:latest  ports:  - containerPort: 80  name: webserver  volumeMounts:  - mountPath: /etc/nginx/nginx.conf  readOnly: true  subPath: nginx.conf  name: nginx-conf  - image: redis  imagePullPolicy: IfNotPresent  name: redis  ports:  - name: redis  containerPort: 6379  protocol: TCP 

When this workload is deployed, the Collector will automatically discover it and identify the specific annotations. After this, two different receivers will be started, each one responsible for each of the target containers.

Collecting Logs from Multiple Target Containers

The annotation-based discovery feature also supports log collection based on the provided annotations. In the example below, we again have a Deployment with a Pod consisting of two different containers, where we want to apply different log collection configurations. We can specify annotations that are scoped to individual container names:

apiVersion: apps/v1 kind: Deployment metadata:  name: busybox-logs-deployment  labels:  app: busybox spec:  replicas: 1  selector:  matchLabels:  app: busybox  template:  metadata:  labels:  app: busybox  annotations:  io.opentelemetry.discovery.logs.lazybox/enabled: "true"  io.opentelemetry.discovery.logs.lazybox/config: |  operators:  - id: container-parser  type: container  - id: some  type: add  field: attributes.tag  value: hints-lazybox  io.opentelemetry.discovery.logs.busybox/enabled: "true"  io.opentelemetry.discovery.logs.busybox/config: |  operators:  - id: container-parser  type: container  - id: some  type: add  field: attributes.tag  value: hints-busybox  spec:  containers:  - name: busybox  image: busybox  args:  - /bin/sh  - -c  - while true; do echo "otel logs from busybox at $(date +%H:%M:%S)" && sleep 5s; done  - name: lazybox  image: busybox  args:  - /bin/sh  - -c  - while true; do echo "otel logs from lazybox at $(date +%H:%M:%S)" && sleep 25s; done 

The above configuration enables two different filelog receiver instances, each applying a unique parsing configuration. This is handy when we know how to parse specific technology logs, such as Apache server access logs.

Combining Both Metrics and Logs Collection

In our third example, we illustrate how to define both metrics and log annotations on the same workload. This allows us to collect both signals from the discovered workload. Below is a Deployment with a Pod consisting of a Redis server and a BusyBox container that performs dummy log writing. We can target annotations to the port and container levels to collect metrics from the Redis server using the Redis receiver, and logs from the BusyBox using the filelog receiver. Here’s how:

apiVersion: apps/v1 kind: Deployment metadata:  name: redis-deployment  labels:  app: redis spec:  replicas: 1  selector:  matchLabels:  app: redis  template:  metadata:  labels:  app: redis  annotations:  io.opentelemetry.discovery.metrics.6379/enabled: "true"  io.opentelemetry.discovery.metrics.6379/scraper: redis  io.opentelemetry.discovery.metrics.6379/config: |  collection_interval: "20s"  timeout: "10s"   io.opentelemetry.discovery.logs.busybox/enabled: "true"  io.opentelemetry.discovery.logs.busybox/config: |  operators:  - id: container-parser  type: container  - id: some  type: add  field: attributes.tag  value: hints  spec:  containers:  - image: redis  imagePullPolicy: IfNotPresent  name: redis  ports:  - name: redis  containerPort: 6379  protocol: TCP  - name: busybox  image: busybox  args:  - /bin/sh  - -c  - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 15s; done 

Explore and analyse data coming from dynamic targets in Elastic

Once the target Pods are discovered and the Collector has started collecting telemetry data from them, we can then explore this data in Elastic. In Discover we can search for Redis and NGINX metrics as well as logs collected from the Busybox container. Here is how it looks like:

Summary

The examples above showcase how users of our OpenTelemetry Collector can take advantage of this new feature — one we played a major role in developing.

For this, we leveraged our years of experience with similar features already supported in Metricbeat, Filebeat, and Elastic-Agent. This makes us extremely happy and confident, as it closes the feature gap between Elastic's specific monitoring agents and the OpenTelemetry Collector — making it even better.

Interested in learning more? Visit the documentation and give it a try by following our EDOT quickstart guide.