DEV Community

Cover image for OTel operator: to simplify observability on kubernetes
Ashok Nagaraj
Ashok Nagaraj

Posted on

OTel operator: to simplify observability on kubernetes

#opentelemetry #kubernetes #observability #instrumentation

OpenTelemetry is a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for understanding, analyzing and debugging purposes.

OTel also promotes the concept of Bring Your Own (BYO) storage backend (eg: elastic stack, AppDynamics, Dynatrace ..) and BYO query (eg: prometheus) by providing common intermediate framework for instrumentation and data collection/transport/transform with OTLP.

OTel kubernetes operator

Image description
This operator provides 2 custom resources:

  1. OTel collector for collection/transform and transport telemetry data
  2. Auto-instrumentation through init container pattern for supported languages (Python, NodeJS and Java at the moment)

Modes of deployment

Image description
The operator(collector CR) can be deployed as:

  1. Deployment (default)
  2. DaemonSet
  3. SideCar injection Note: Multiple modes can co-exist based on the architecture (eg: side-car collectors pushing to a central Deployment aggregator) Official documentation
Installation 
# Pre-req: kubernetes cluster with cert-manager enabled # Deployment based $ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml namespace/opentelemetry-operator-system created customresourcedefinition.apiextensions.k8s.io/instrumentations.opentelemetry.io created ... deployment.apps/opentelemetry-operator-controller-manager created $ kubectl get crds | grep opentel instrumentations.opentelemetry.io 2022-11-14T06:14:32Z opentelemetrycollectors.opentelemetry.io 2022-11-14T06:14:32Z $ kubectl get deployments.apps -n opentelemetry-operator-system NAME READY UP-TO-DATE AVAILABLE AGE opentelemetry-operator-controller-manager 1/1 1 1 56s # Create a collector like below $ kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel-col spec: config: | receivers: otlp: protocols: grpc: http: processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 batch: send_batch_size: 10000 timeout: 10s exporters: logging: service: pipelines: traces: receivers: [otlp] processors: [] exporters: [logging] EOF opentelemetrycollector.opentelemetry.io/otel-col created
Enter fullscreen mode Exit fullscreen mode

Auto-instrumentation

  1. Create an instrumentation object 
$ kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: exporter: endpoint: http://otel-collector:4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "0.25" EOF instrumentation.opentelemetry.io/my-instrumentation created $ kubectl get otelinst NAME AGE ENDPOINT SAMPLER SAMPLER ARG my-instrumentation 35s http://otel-collector:4317 parentbased_traceidratio 0.25 ku
Enter fullscreen mode Exit fullscreen mode
  1. Enabling instrumentation is done through annotations 
 metadata: annotations: instrumentation.opentelemetry.io/inject-python: "true" # or inject-java or inject-nodejs
Enter fullscreen mode Exit fullscreen mode

Possible values are true, false(no instrumentation), my-instrumentation (custom instrumentation) or my-namespace/my-instrumentation (instrumentation defined in my-namespace).

Further tuning on which containers to instrument can be done with annotation instrumentation.opentelemetry.io/container-names: a,b,c...

When side-car based instrumentation is to be done, use the annotation sidecar.opentelemetry.io/inject: "true"
To use 3rd-party instrumentation, check this link

Demo

My partially working gist so-far is here

$ kubectl apply -f . opentelemetrycollector.opentelemetry.io/sidecar-collector created deployment.apps/flask-example created instrumentation.opentelemetry.io/python-instrumentation created service/flask-example created $ kubectl get opentelemetrycollectors.opentelemetry.io,instrumentations.opentelemetry.io NAME MODE VERSION AGE opentelemetrycollector.opentelemetry.io/sidecar-collector sidecar 0.63.1 80s NAME AGE ENDPOINT SAMPLER SAMPLER ARG instrumentation.opentelemetry.io/python-instrumentation 79s http://localhost:4317 parentbased_traceidratio 0.25 $ kubectl get deployments.apps NAME READY UP-TO-DATE AVAILABLE AGE flask-example 1/1 1 1 49s $ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].image}" ashoka007/flask-example:0.1 ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.63.1% $ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].name}" flask-example otc-container% # otc-container is the local side-car collector
Enter fullscreen mode Exit fullscreen mode

Here is a full-fledged working demo and a detailed blog - https://isitobservable.io/open-telemetry/how-to-observe-your-kubernetes-cluster-with-opentelemetry

Top comments (0)