Istio Webhook Management [Experimental]

5 minute read

Istio has two webhooks: Galley and the sidecar injector. By default, these webhooks manage their own configurations. From a security perspective, this default behavior is not recommended because a compromised webhook could then conduct privilege escalation attacks.

This task shows how to use the new istioctl x post-install webhook command to securely manage the configurations of the webhooks.

Getting started

Install Istio with DNS certificates configured and global.operatorManageWebhooks set to true.

$ cat <<EOF > ./istio.yaml apiVersion: install.istio.io/v1alpha2 kind: IstioControlPlane spec: values: global: operatorManageWebhooks: true certificates: - secretName: dns.istio-galley-service-account dnsNames: [istio-galley.istio-system.svc, istio-galley.istio-system] - secretName: dns.istio-sidecar-injector-service-account dnsNames: [istio-sidecar-injector.istio-system.svc, istio-sidecar-injector.istio-system] EOF $ istioctl manifest apply -f ./istio.yaml

Install jq for JSON parsing.

Check webhook certificates

To display the DNS names in the webhook certificates of Galley and the sidecar injector, you need to get the secret from Kubernetes, parse it, decode it, and view the text output with the following commands:

$ kubectl get secret dns.istio-galley-service-account -n istio-system -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -in - -text -noout $ kubectl get secret dns.istio-sidecar-injector-service-account -n istio-system -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -in - -text -noout

The output from the above commands should include the DNS names of Galley and the sidecar injector, respectively:

X509v3 Subject Alternative Name: DNS:istio-galley.istio-system.svc, DNS:istio-galley.istio-system

X509v3 Subject Alternative Name: DNS:istio-sidecar-injector.istio-system.svc, DNS:istio-sidecar-injector.istio-system

Enable webhook configurations

To generate the MutatingWebhookConfiguration and ValidatingWebhookConfiguration configuration files, run the following command.
```
$ istioctl manifest generate > istio.yaml 
```

Open the istio.yaml configuration file, search for kind: MutatingWebhookConfiguration and save the MutatingWebhookConfiguration of the sidecar injector to sidecar-injector-webhook.yaml. The following is a MutatingWebhookConfiguration in an example istio.yaml.

apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: istio-sidecar-injector labels: app: sidecarInjectorWebhook release: istio webhooks: - name: sidecar-injector.istio.io clientConfig: service: name: istio-sidecar-injector namespace: istio-system path: "/inject" caBundle: "" rules: - operations: [ "CREATE" ] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] failurePolicy: Fail namespaceSelector: matchLabels: istio-injection: enabled

Open the istio.yaml configuration file, search for kind: ValidatingWebhookConfiguration and save the ValidatingWebhookConfiguration of Galley to galley-webhook.yaml. The following is a ValidatingWebhookConfiguration in an example istio.yaml (only a part of the configuration is shown to save space).

apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration metadata: name: istio-galley labels: app: galley release: istio istio: galley webhooks: - name: pilot.validation.istio.io clientConfig: service: name: istio-galley namespace: istio-system path: "/admitpilot" caBundle: "" rules: - operations: - CREATE - UPDATE apiGroups: - config.istio.io ... SKIPPED failurePolicy: Fail sideEffects: None

Verify that there are no existing webhook configurations for Galley and the sidecar injector. The output of the following two commands should not contain any configurations for Galley and the sidecar injector.
```
$ kubectl get mutatingwebhookconfiguration $ kubectl get validatingwebhookconfiguration 
```
If there are existing webhook configurations (e.g., from a previous Istio deployment) for Galley and the sidecar injector, delete them using the following commands. Before running these commands, replace the webhook configuration names in the commands with the actual webhook configuration names of Galley and the sidecar injector in your cluster.
```
$ kubectl delete mutatingwebhookconfiguration SIDECAR-INJECTOR-WEBHOOK-CONFIGURATION-NAME $ kubectl delete validatingwebhookconfiguration GALLEY-WEBHOOK-CONFIGURATION-NAME 
```

Use istioctl to enable the webhook configurations:

$ istioctl experimental post-install webhook enable --webhook-secret dns.istio-galley-service-account \ --namespace istio-system --validation-path galley-webhook.yaml \ --injection-path sidecar-injector-webhook.yaml

To check that the sidecar injector webhook is working, verify that the webhook injects a sidecar container into an example pod with the following commands:

$ kubectl create namespace test-injection $ kubectl label namespaces test-injection istio-injection=enabled $ kubectl run --generator=run-pod/v1 --image=nginx nginx-app --port=80 -n test-injection $ kubectl get pod -n test-injection

The output from the get pod command should show the following. The 2/2 value means that the webhook injected a sidecar into the example pod:

NAME READY STATUS RESTARTS AGE nginx-app 2/2 Running 0 10s

Check that the validation webhook is working:

$ kubectl create namespace test-validation $ kubectl apply -n test-validation -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: invalid-gateway spec: selector: # DO NOT CHANGE THESE LABELS # The ingressgateway is defined in install/kubernetes/helm/istio/values.yaml # with these labels istio: ingressgateway EOF

The output from the gateway creation command should show the following output. The error in the output indicates that the validation webhook checked the gateway’s configuration YAML file:

Error from server: error when creating "invalid-gateway.yaml": admission webhook "pilot.validation.istio.io" denied the request: configuration is invalid: gateway must have at least one server

Show webhook configurations

If you named the sidecar injector’s configuration istio-sidecar-injector and named Galley’s configuration istio-galley-istio-system, use the following command to show the configurations of these two webhooks:
```
$ istioctl experimental post-install webhook status --validation-config=istio-galley-istio-system --injection-config=istio-sidecar-injector 
```
If you named the sidecar injector’s configuration istio-sidecar-injector, use the following command to show the configuration of the sidecar injector:
```
$ istioctl experimental post-install webhook status --validation=false --injection-config=istio-sidecar-injector 
```
If you named Galley’s configuration istio-galley-istio-system, show Galley’s configuration with the following command:
```
$ istioctl experimental post-install webhook status --injection=false --validation-config=istio-galley-istio-system 
```

Disable webhook configurations

If you named the sidecar injector’s configuration istio-sidecar-injector and named Galley’s configuration istio-galley-istio-system, use the following command to disable the configurations of these two webhooks:
```
$ istioctl experimental post-install webhook disable --validation-config=istio-galley-istio-system --injection-config=istio-sidecar-injector 
```
If you named the sidecar injector’s configuration istio-sidecar-injector, disable the webhook with the following command:
```
$ istioctl experimental post-install webhook disable --validation=false --injection-config=istio-sidecar-injector 
```

If you named Galleys’s configuration istio-galley-istio-system, disable the webhook with the following command:

$ istioctl experimental post-install webhook disable --injection=false --validation-config=istio-galley-istio-system

Cleanup

You can run the following command to delete the resources created in this tutorial.

$ kubectl delete ns test-injection test-validation $ kubectl delete -f galley-webhook.yaml $ kubectl delete -f sidecar-injector-webhook.yaml