Customizing Istio Metrics with Telemetry API

Telemetry API has been in Istio as a first-class API for quite sometime now. Previously, users had to configure metrics in the telemetry section of the Istio configuration.

This task shows you how to customize the metrics that Istio generates with Telemetry API.

Before you begin

Install Istio in your cluster and deploy an application.

Telemetry API can not work together with EnvoyFilter. For more details please checkout this issue.

• Starting with Istio version 1.18, the Prometheus EnvoyFilter will not be installed by default, and instead meshConfig.defaultProviders is used to enable it. Telemetry API should be used to further customize the telemetry pipeline.

• For versions of Istio before 1.18, you should install with the following IstioOperator configuration:

  apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: values: telemetry: enabled: true v2: enabled: false

Override metrics

The metrics section provides values for the metric dimensions as expressions, and allows you to remove or override the existing metric dimensions. You can modify the standard metric definitions using tags_to_remove or by re-defining a dimension.

1. Remove grpc_response_status tags from REQUEST_COUNT metric

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: remove-tags namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - match: mode: CLIENT_AND_SERVER metric: REQUEST_COUNT tagOverrides: grpc_response_status: operation: REMOVE

2. Add custom tags for REQUEST_COUNT metric

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: custom-tags namespace: istio-system spec: metrics: - overrides: - match: metric: REQUEST_COUNT mode: CLIENT tagOverrides: destination_x: value: upstream_peer.labels['app'].value - match: metric: REQUEST_COUNT mode: SERVER tagOverrides: source_x: value: downstream_peer.labels['app'].value providers: - name: prometheus

Disable metrics

1. Disable all metrics by following configuration:

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: remove-all-metrics namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: CLIENT_AND_SERVER metric: ALL_METRICS

2. Disable REQUEST_COUNT metrics by following configuration:

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: remove-request-count namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: CLIENT_AND_SERVER metric: REQUEST_COUNT

3. Disable REQUEST_COUNT metrics for client by following configuration:

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: remove-client namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: CLIENT metric: REQUEST_COUNT

4. Disable REQUEST_COUNT metrics for server by following configuration:

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: remove-server namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: SERVER metric: REQUEST_COUNT

Verify the results

Send traffic to the mesh. For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your web browser or issue the following command:

$ curl "http://$GATEWAY_URL/productpage"

Use the following command to verify that Istio generates the data for your new or modified dimensions:

$ istioctl x es "$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')" -oprom | grep istio_requests_total | grep -v TYPE |grep -v 'reporter="destination"'

$ istioctl x es "$(kubectl get pod -l app=details -o jsonpath='{.items[0].metadata.name}')" -oprom | grep istio_requests_total

For example, in the output, locate the metric istio_requests_total and verify it contains your new dimension.