Securing the metrics endpoint

ECK

If you're using ECK 2.16 or earlier, then after you enable your metrics endpoint, you need to secure it. To enable RBAC and TLS on the metrics endpoint, follow the instructions in the following sections depending on whether you installed ECK through the Helm chart or the manifests.

The ECK operator metrics endpoint is secured by default beginning in version 3.0.0. If you're using ECK 3.0.0 or later, then you might want to use the information on this page to provide your own TLS certificate.

Using the operator Helm chart

If you installed ECK through the Helm chart commands listed in Install using a Helm chart, you can set config.metrics.secureMode.enabled to true and both RBAC and TLS/HTTPs will be enabled for the metrics endpoint.

Using your own TLS certificate for the metrics endpoint when using the Helm chart

By default, a self-signed certificate will be generated for use by the metrics endpoint. If you want to use your own TLS certificate for the metrics endpoint, you can provide the config.metrics.secureMode.tls.certificateSecret to the Helm chart. The certificateSecret should be the name of an existing Kubernetes Secret that contains both the TLS certificate and the TLS private key. The following keys are supported within the secret:

tls.crt: The PEM-encoded TLS certificate
tls.key: The PEM-encoded TLS private key

The easiest way to create this secret is to use the kubectl create secret tls command. For example:

  kubectl create secret tls eck-metrics-tls-certificate -n elastic-system --cert=/path/to/tls.crt --key=/path/to/tls.key  

Providing this secret is sufficient to use your own certificate if it is from a trusted Certificate Authority. If the certificate is not signed by a trusted CA and you are using Prometheus to scrape the metrics, you have the following options:

Disable TLS verification.
- Set serviceMonitor.insecureSkipVerify to true to disable TLS validation in the ServiceMonitor generated by the eck-operator Helm chart.
Provide the Certificate Authority to Prometheus.
- Set serviceMonitor.insecureSkipVerify to false to enable TLS validation.
- Set serviceMonitor.caSecret to the name of an existing Kubernetes secret within the Prometheus namespace that contains the CA in PEM format in a file called ca.crt.
- Set the spec.secrets field of the Prometheus custom resource, or prometheus.prometheusSpec.secrets when using the Helm chart such that the CA secret is mounted into the Prometheus pod at serviceMonitor.caMountDirectory (assuming you are using the Prometheus operator). See the ECK Helm chart values file for more information.

Refer to Prometheus requirements for more information on creating the CA secret.

Using the operator manifests

If you installed ECK through using the manifests using the commands listed in Install ECK using the YAML manifests, some additional changes are required to enable secure metrics.

Enable the metrics port in the ConfigMap and set the metrics-secure setting to true.

  cat <<EOF | kubectl apply -f - kind: ConfigMap apiVersion: v1 metadata: name: elastic-operator namespace: elastic-system data: eck.yaml: |- log-verbosity: 0 metrics-port: 8081 metrics-host: 0.0.0.0 metrics-secure: true container-registry: docker.elastic.co max-concurrent-reconciles: 3 ca-cert-validity: 8760h ca-cert-rotate-before: 24h cert-validity: 8760h cert-rotate-before: 24h disable-config-watch: false exposed-node-labels: [topology.kubernetes.io/.*,failure-domain.beta.kubernetes.io/.*] set-default-security-context: auto-detect kube-client-timeout: 60s elasticsearch-client-timeout: 180s disable-telemetry: false distribution-channel: all-in-one validate-storage-class: true enable-webhook: true webhook-name: elastic-webhook.k8s.elastic.co webhook-port: 9443 operator-namespace: elastic-system enable-leader-election: true elasticsearch-observation-interval: 10s ubi-only: false EOF  

Add an additional ClusterRole and ClusterRoleBinding for the ECK operator.

  cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: elastic-operator-metrics-auth-role rules: - apiGroups: - authentication.k8s.io resources: - tokenreviews verbs: - create - apiGroups: - authorization.k8s.io resources: - subjectaccessreviews verbs: - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: elastic-operator-metrics-auth-rolebinding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: elastic-operator-metrics-auth-role subjects: - kind: ServiceAccount name: elastic-operator namespace: elastic-system EOF  

Add a Service to expose the metrics endpoint.

  cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: labels: control-plane: elastic-operator app.kubernetes.io/component: metrics name: elastic-operator-metrics namespace: elastic-system spec: ports: - name: https port: 8080 protocol: TCP targetPort: metrics selector: control-plane: elastic-operator EOF  

If you're using the Prometheus operator, add a ServiceMonitor to allow Prometheus to scrape the metrics endpoint.

  cat <<EOF | kubectl apply -f - apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: elastic-operator namespace: elastic-system spec: namespaceSelector: matchNames: - elastic-system selector: matchLabels: control-plane: elastic-operator app.kubernetes.io/component: metrics endpoints: - port: https path: /metrics scheme: https interval: 30s tlsConfig: insecureSkipVerify: true bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token EOF  

Using your own TLS certificate for the metrics endpoint when using the manifests

By default a self-signed certificate will be generated for use by the metrics endpoint. If you want to use your own TLS certificate for the metrics endpoint you will need to follow the previous instructions to enable secure metrics as well as the following steps:

Create a Secret containing the TLS certificate and TLS private key. The following keys are supported within the secret:
- tls.crt - The PEM-encoded TLS certificate
- tls.key - The PEM-encoded TLS private key
The easiest way to create this secret is to use the kubectl create secret tls command. For example:
```
 kubectl create secret tls my-tls-secret -n elastic-system --cert=/path/to/tls.crt --key=/path/to/tls.key 
```

Patch the StatefulSet to include the tls.crt and tls.key as a volume and mount it into the manager container.

  kubectl patch sts -n elastic-system elastic-operator --patch-file=/dev/stdin <<-EOF spec: template: spec: containers: - name: manager volumeMounts: - mountPath: "/tmp/k8s-metrics-server/serving-certs" name: tls-certificate readOnly: true volumes: - name: conf configMap: name: elastic-operator - name: cert secret: defaultMode: 420 secretName: elastic-webhook-server-cert - name: tls-certificate secret: defaultMode: 420 secretName: eck-metrics-tls-certificate EOF  

If you're mounting the TLS secret to a different directory, the metrics-cert-dir setting in the operator configuration has to be adjusted accordingly.

If required, patch the ServiceMonitor. This is required if you are adjusting the insecureSkipVerify field to false.

  kubectl patch servicemonitor -n elastic-system elastic-operator --patch-file=/dev/stdin <<-EOF spec: endpoints: - port: https path: /metrics scheme: https interval: 30s tlsConfig: insecureSkipVerify: false caFile: /etc/prometheus/secrets/{secret-name}/ca.crt serverName: elastic-operator-metrics.elastic-system.svc bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token EOF  

See Prometheus requirements for more information on creating the CA secret.