8

I am finding all sorts of walkthroughs on how to add certificates to be used in the pods themselves, but I can't seem to find info on how to setup Kubernetes to allow a self-signed cert for pulling images from a Harbor instance running inside the cluster. I have the ca cert imported to the system's trusted certs (system is running Ubuntu 18.04), but I am guessing Kubernetes uses it's own trusted certs store somewhere, similar to how Java ignores the system's trusted certs and relies on keystore files?

Edit To be more specific, what I am trying to do is deploy a custom Docker image stored in my Harbor instance that is running in my Kubernetes cluster. I have the certs for Harbor imported into my host systems OS, and I can run:

docker login <url_to_harbor> docker pull <url_to_harbor>/library/custom/image:latest

and it works fine from CLI, but if I try to create a deployment yaml like so:

apiVersion: apps/v1 kind: Deployment metadata: name: custom-deployment spec: replicas: 1 selector: matchLabels: app: custom template: metadata: labels: app: custom spec: containers: - name: custom image: <url_to_harbor>/library/custom/image:latest ...

and run kubectl apply -f custom-deploy.yaml I get the following error when I get pods:

custom-deployment-6ff68947f6-8jj2p 0/1 ImagePullBackOff 0 13s

And if I get a description on the failed pod I see:

 Warning Failed 18s kubelet, node3 Failed to pull image "<url_to_harbor>/library/custom/image:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://<url_to_harbor>/v2/: x509: certificate signed by unknown authority Warning Failed 18s kubelet, node3 Error: ErrImagePull

I have restarted the host machine since I imported the CA certs. I added the certs by moving the root CA cert file into /usr/local/share/ca-certificates and then running sudo update-ca-certificates.

Improve this question
4
  • 1
    Have you restarted the container engine after importing the ca cert into kubernetes nodes? What's the result of docker pull <your image> directly inside one of your kubernetes nodes? Commented Jun 7, 2020 at 8:25
  • @rgio I added some addition details to the original post. Commented Jun 7, 2020 at 14:49
  • 2
    You talked about a single host machine: actually you have to add ca cert to every kubernetes node: all workers and masters. What kind of kubernetes cluster do you have? Commented Jun 7, 2020 at 17:05
  • I have 4 servers, one as the master and 3 additional nodes. I only added the cert to the master. Let me try to add the certs to the other nodes and restart. Commented Jun 8, 2020 at 10:59

2 Answers 2

Reset to default
6

The method of importing CA cert is correct, but you have to do that on every cluster node: all masters and workers. Kubernetes simply relies on CAs installed on underlying operating system.

Improve this answer
0
3

If you're in a situation where you may not want to install the CA cert on every node, you can also run a DaemonSet which configures the ca-certificate. Note, this will only work if Kubernetes is configured with the containerd runtime.

apiVersion: v1 kind: ConfigMap metadata: name: trusted-ca namespace: kube-system data: ca.crt: |+ -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- --- apiVersion: v1 kind: ConfigMap metadata: name: setup-script namespace: kube-system data: setup.sh: | echo "$TRUSTED_CERT" > /usr/local/share/ca-certificates/ca.crt && update-ca-certificates && systemctl restart containerd --- apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-system name: node-custom-setup labels: k8s-app: node-custom-setup spec: selector: matchLabels: k8s-app: node-custom-setup template: metadata: labels: k8s-app: node-custom-setup spec: hostPID: true hostNetwork: true initContainers: - name: init-node command: ["nsenter"] args: ["--mount=/proc/1/ns/mnt", "--", "sh", "-c", "$(SETUP_SCRIPT)"] image: debian env: - name: TRUSTED_CERT valueFrom: configMapKeyRef: name: trusted-ca key: ca.crt - name: SETUP_SCRIPT valueFrom: configMapKeyRef: name: setup-script key: setup.sh securityContext: privileged: true containers: - name: wait image: k8s.gcr.io/pause:3.1

Source: http://hypernephelist.com/2021/03/23/kubernetes-containerd-certificate.html

These instructions were written prior to the implementation of a host registry directory in cri in containerd 1.5, so if running a new version, it may be possible to configure without restarts. See Registry Configuration - Introduction.

Finally, if you're running on a cloud managed cluster, they may provide a custom configuration for managing CA Certificates. E.g., AKS - Custom CA

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.