DEV Community

Cover image for A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts
AWS Community Builders profile image Sunny Bhambhani
Sunny Bhambhani for AWS Community Builders

Posted on

A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts

#kubernetes #devops #linux #security

Kubernetes is everywhere now a days, so does the container images and fetching the images from a private registry is a norm because of N number of reasons including security, that being the topmost.

Recap

Just to give you a heads up before we dive further, let's assume I fire kubectl create deployment nginx --image nginx what do you think happens?

  1. Using kubeconfig, kubectl first authenticates with the API server.
  2. It then constructs a deployment object using the values which we provided like deployment name as nginx, image to be used as nginx and some default values and then sends a request to the API server.
  3. API server then validates the request and the object definition.
  4. This is further stored in etcd datastore.
  5. Then the controller manager specifically deployment controller comes into the picture which creates a replica set.
  6. Then the replica set creates a pod (this is where our nginx image will be used).
  7. Then the schedular assigns the pod to a node, where it will run based on multiple conditions. If interested please check out my previous article on what all things happens in the background: Resource management in Kubernetes
  8. Once the node is identified, the kubelet running on the node is responsible for the management of the pod.
  9. Once the pod is scheduled on a specific node, the kubelet looks for the image to be used and if it is already present.

if that is present locally, it will use it.

If not, it pulls the image from the registry.

If the registry is public, there are no issues, it just pulls the image, runs the pods, and we are done.

When imagePullSecrets are used?

  • Assume the registry is private, this is where imagePullSecrets come into play.
  • Kubernetes will need the secrets to authenticate the registry, so that it can pull the image.

How it works?

  • Firstly, we need to create a secret which contains the credentials and some details of our docker registry like server-name, username, password, email, etc.
  • It can be created both imperative as well as declarative way. 
$ k get secrets -o yaml dockersec apiVersion: v1 data: .dockerconfigjson: ENCODED JSON OBJECT kind: Secret metadata: name: dockersec namespace: default type: kubernetes.io/dockerconfigjson
Enter fullscreen mode Exit fullscreen mode 
$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD
Enter fullscreen mode Exit fullscreen mode
  • Quick note: The secret must be of a type kubernetes.io/dockerconfigjson or docker-registry based on the way you choose to create the secret.
  • Once the secret is created, it must be attached to a pod so that it can use it.
  • Just add below content in your pod/deployment/statefulset yaml files, the name of the secret for me is dockersec but you can change based on your requirements. 
spec: imagePullSecrets: - name: dockersec
Enter fullscreen mode Exit fullscreen mode
  • Before pulling the image, the kubelet checks if the image is from a private registry. If so, it looks for imagePullSecrets in the spec.
  • Without proper imagePullSecrets, the image pull will fail with ImagePullBackOff.

Problem

  • Now assume you have hundreds of YAML files and hundreds of Kubernetes Object where we have to use this imagePullSecrets.
  • Just for the sake of doing once, you thought to do it manually but just imagine there was a change in the name of the Kubernetes secret.
  • Maybe because the organization now decides to streamline the naming convention of all the Kubernetes objects based on environments like prod/preprod/staging/etc.
    • Or your org is now moving to a different registry or a different provider.
    • Or the application is getting re-factored based on different namespaces or functions.
    • There can be N number of reasons and doing all these changes again is a tricky and tiresome task.

One way to look at it is, use of helm charts which can help us to manage all the Kubernetes objects and this kind of a change can be pretty straight forward provided the helm charts are structured correctly and using values file we can just update the name of the secret and then can upgrade the chart, and we are done.

But if in case we are not using and package manager like helm and we are just working with vanilla YAML files then for that as well there is a neat way to do it.

The NEAT way!

We can attach the imagePullSecrets with the serviceaccounts and where-ever the serviceaccount is used, it will automatically populate the imagePullSecrets in the respective object like Pods.

  • Below is the current status of my Kubernetes objects, no secrets, a default serviceaccount and no imagePullSecrets updated in the pod. 
$ k get secrets No resources found in default namespace. $ k get sa NAME SECRETS AGE default 0 33m $ k get pods NAME READY STATUS RESTARTS AGE nginx-65fd88fc88-8xjc9 0/1 ImagePullBackOff 0 32m
Enter fullscreen mode Exit fullscreen mode
  • Create the docker-registry secret 
$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD secret/dockersec created $ k get secret NAME TYPE DATA AGE dockersec kubernetes.io/dockerconfigjson 1 3s
Enter fullscreen mode Exit fullscreen mode
  • Patch the default serviceaccount to have the imagePullSecrets. It can be any serviceaccount based on your requirements. 
$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "dockersec"}]}' serviceaccount/default patched $ k get sa NAME SECRETS AGE default 0 35m
Enter fullscreen mode Exit fullscreen mode
  • Or you can manually edit the serviceaccount as well and add below content. 
$ k get sa default -o yaml apiVersion: v1 imagePullSecrets: # ADD - name: dockersec # ADD kind: ServiceAccount metadata: name: default namespace: default
Enter fullscreen mode Exit fullscreen mode
  • To see this in action now, delete the pod which is in failed state. 
$ k delete pod nginx-65fd88fc88-8xjc9 --force
Enter fullscreen mode Exit fullscreen mode
  • Check the events of the pod, now it was successfully able to pull the image from the private registry. 
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 65s default-scheduler Successfully assigned default/nginx-65fd88fc88-w8mph to minikube Normal Pulling 64s kubelet Pulling image "sunnybhambhani/test:v1" Normal Pulled 37s kubelet Successfully pulled image "sunnybhambhani/test:v1" in 30.288s (30.288s including waiting). Image size: 567185619 bytes. Normal Created 21s (x3 over 36s) kubelet Created container: test Normal Started 21s (x3 over 36s) kubelet Started container test
Enter fullscreen mode Exit fullscreen mode
  • Interesting bit is imagePullSecrets are automatically added to the object. 
$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i serviceaccountname serviceAccountName: default $ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i imagepullse -A 1 imagePullSecrets: - name: dockersec
Enter fullscreen mode Exit fullscreen mode

Bonus Tip

  • In case the pod is not in running status, it can be due to multiple reasons.
  • Just describe the pod to see what is happening in the background.
  • kubectl describe pod POD_NAME | awk '/^Events:/ {found=1; next} found', it looks for the line starting with Events, once found prints the remaining lines. 
$ kubectl describe pod nginx-65fd88fc88-8xjc9 | awk '/^Events:/ {found=1; next} found' Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7m21s default-scheduler Successfully assigned default/nginx-65fd88fc88-8xjc9 to minikube Normal Pulling 4m17s (x5 over 7m20s) kubelet Pulling image "REGISTRY/test:v1" Warning Failed 4m14s (x5 over 7m17s) kubelet Failed to pull image "REGISTRY/test:v1": Error response from daemon: pull access denied for REGISTRY/test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied Warning Failed 4m14s (x5 over 7m17s) kubelet Error: ErrImagePull Warning Failed 2m41s (x19 over 7m16s) kubelet Error: ImagePullBackOff Normal BackOff 2m20s (x21 over 7m16s) kubelet Back-off pulling image "REGISTRY/test:v1"
Enter fullscreen mode Exit fullscreen mode
  • Here the logs clearly state, requested access to the resource is denied and in-turn the status of the pod is ImagePullBackOff. 
$ k get pods NAME READY STATUS RESTARTS AGE nginx-65fd88fc88-8xjc9 0/1 ImagePullBackOff 0 76s
Enter fullscreen mode Exit fullscreen mode

References:

PS: This works with any flavor of Kubernetes: minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.

I hope this will be helpful, feel free to add your thoughts and experiences, Happy Learning! 😄

Top comments (1)

Collapse
 
__1ef3200 profile image
Володимир Смаглюк •

Thanks to author for the effort! I didn't know we can use ServiceAccount to propagate the imagePullSecret to pod spec. It's nice to know.

At first, I don't understand how the ServiceAccount solves the "Problem: Namespace" you've mentioned about.

Or the application is getting re-factored based on different namespaces or functions.

ServiceAccount is a namespaced resource. You still need to the ServiceAccount per namespace, and update each Pod's spec to use the ServiceAccount. But than I realised that the the point is to grant those permissions to the default ServiceAccount, so any pod in the namespace which will be using this ServiceAccount will have the required imagePullSecret block.

I hope, someone will find a way to propagate it on global level.