Label metrics

Labels let you differentiate the characteristics of a metric. In addition to collecting metrics, the MonitoringTarget custom resource lets you label metrics for optional observability features:

For more information on metrics relabeling, see https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/.

Before you begin

To get the permissions that you need to manage MonitoringTarget custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one of the associated MonitoringTarget roles.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for this resource in an organization or a project. For more information, see Prepare IAM permissions.

Send metrics to another project

Container authors can code specific metrics to be sent to a different project, even to projects they don't own. To send a metric to a specific project, add the _gdch_project label to the metric in your code, setting its value to the destination project name.

Alternatively, use the MonitoringTarget custom resource to send all collected metric data to another project. Set the _gdch_project label as a target label in the metricsRelabelings field, with the new project name as the replacement value.

The following code sample shows how to set the _gdch_project target label in the MonitoringTarget custom resource to send metrics to the project named another-project-name:

apiVersion: monitoring.gdc.goog/v1 kind: MonitoringTarget metadata:  # Choose the same namespace as the workload pods.  namespace: PROJECT_NAMESPACE  name: MONITORING_TARGET_NAME spec:  [...]  podMetricsEndpoints:  [...]  metricsRelabelings:  - action: replace  targetLabel: _gdch_project  replacement: another-project-name  [...]

Replace the following:

  • PROJECT_NAMESPACE: your project namespace.
  • MONITORING_TARGET_NAME: the name of the MonitoringTarget definition file.

Add new labels to metrics

Labels help with observability and monitoring by identifying metric characteristics. You can add labels that the scraped containers don't initially expose. In the MonitoringTarget custom resource, set the new label as a target and provide a replacement value. This key-value pair is added to all metrics the resource collects.

The following code sample shows how to set the my_new_label target label and configure its value to my_label_value in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1 kind: MonitoringTarget metadata:  # Choose the same namespace as the workload pods.  namespace: PROJECT_NAMESPACE  name: MONITORING_TARGET_NAME spec:  [...]  podMetricsEndpoints:  [...]  metricsRelabelings:  - action: replace  targetLabel: my_new_label  replacement: my_label_value  [...]

Replace the following:

  • PROJECT_NAMESPACE: your project namespace.
  • MONITORING_TARGET_NAME: the name of the MonitoringTarget definition file.

Rename a metric

You can rename a container's metric using the MonitoringTarget resource. Set the original metric name and its replacement in the metricsRelabelings field. The original name is treated as a regular expression (regex) and is replaced by the new name.

The following code sample shows how to replace the cpu_usage metric name (regex) with cpu_usage_new_name in the MonitoringTarget custom resource:

apiVersion: monitoring.gdc.goog/v1 kind: MonitoringTarget metadata:  # Choose the same namespace as the workload pods.  namespace: PROJECT_NAMESPACE  name: MONITORING_TARGET_NAME spec:  [...]  podMetricsEndpoints:  [...]  metricsRelabelings:  - action: replace  regex: cpu_usage  replacement: cpu_usage_new_name  sourceLabels: [__name__]  targetLabel: __name__  [...]

Replace the following:

  • PROJECT_NAMESPACE: your project namespace.
  • MONITORING_TARGET_NAME: the name of the MonitoringTarget definition file.