ImagePullBackOff err while pulling docker image in different project in Google Cloud

Question

I am trying to pull a private image from Artifact Registry repo in Google Cloud from a kubernetes cluster running in a different Google Cloud project using kubectl.

kubernetes version 1.20.15-gke.1000 

The service account for the kubernetes has already been given artifactregistry.reader and storageobject.viewer permissions as the image is in a different project from the kubernetes service account

I apply the below yaml to the kubectl command.

kubectl apply -f proxy_with_workload_identity.yaml apiVersion: apps/v1 kind: Deployment metadata: name: test-app spec: selector: matchLabels: app: app-project strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: labels: app: app-project spec: containers: - env: - name: DB_USER valueFrom: secretKeyRef: key: username name: db-credentials - name: DB_PASS valueFrom: secretKeyRef: key: password name: db-credentials - name: DB_NAME value: postgres image: "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1" name: app-project ports: - containerPort: 9376 protocol: TCP - command: - /cloud_sql_proxy - "-instances=demo-dev:us-central1:1-sql-1=tcp:5432" image: "gcr.io/cloudsql-docker/gce-proxy:latest" name: cloud-sql-proxy resources: requests: cpu: 200m memory: 32Mi securityContext: runAsNonRoot: true serviceAccountName: testapp 

The cloud-sql-proxy image is getting pulled and the container is running , but the image in the private-repository is not getting pulled "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1"

when i check the pods i am shown this error:

code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1:v1": failed to resolve reference "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden 

Can anyone tell me how to solve this?

Answer 1

I had a similar issue this week. Let's assume that your GKE and service account are in project A, and the artifact registry is in project B.

1. Make sure you give the artifactregistry.reader role in project B (not A)
2. Make sure you give this role to the correct service account. The service account that is pulling the image is the Node's service account, not the Pod's service account. In my case, I was using GKE Autopilot that was configured via TerraForm. Because of a bug in the Terraform module, the service account used by the node is default (which is the default compute service account, e.g. <project_number>@cloudservices.gserviceaccount.com)

You can check which service account your nodepool is running with the gcloud container clusters describe ... command.

I hope this helps!

Answer 2

Using images from other projects in your configuration

This page describes how to configure your project so that Deployment Manager can create Compute Engine virtual machine instances using operating system images that belong to another project.

I hope you can use this to update the registry between projects.