ImagePullBackOff err while pulling docker image in different project in Google Cloud

I am trying to pull a private image from Artifact Registry repo in Google Cloud from a kubernetes cluster running in a different Google Cloud project using kubectl.

kubernetes version 1.20.15-gke.1000 

The service account for the kubernetes has already been given artifactregistry.reader and storageobject.viewer permissions as the image is in a different project from the kubernetes service account

I apply the below yaml to the kubectl command.

kubectl apply -f proxy_with_workload_identity.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app
spec:
  selector:
    matchLabels:
      app: app-project
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: app-project
    spec:
      containers:
        - env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: db-credentials
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  key: password
                  name: db-credentials
            - name: DB_NAME
              value: postgres
          image: "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1" 
          name: app-project
          ports:
            - containerPort: 9376
              protocol: TCP
        - command:
            - /cloud_sql_proxy 
            - "-instances=demo-dev:us-central1:1-sql-1=tcp:5432"
          image: "gcr.io/cloudsql-docker/gce-proxy:latest"
          name: cloud-sql-proxy
          resources:
            requests:
              cpu: 200m
              memory: 32Mi
          securityContext:
            runAsNonRoot: true
      serviceAccountName: testapp

The cloud-sql-proxy image is getting pulled and the container is running , but the image in the private-repository is not getting pulled "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1"

when i check the pods i am shown this error:

code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1:v1": failed to resolve reference "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

Can anyone tell me how to solve this?

Using images from other projects in your configuration

This page describes how to configure your project so that Deployment Manager can create Compute Engine virtual machine instances using operating system images that belong to another project.

I hope you can use this to update the registry between projects.

I had a similar issue this week. Let's assume that your GKE and service account are in project A, and the artifact registry is in project B.

1. Make sure you give the artifactregistry.reader role in project B (not A)
2. Make sure you give this role to the correct service account. The service account that is pulling the image is the Node's service account, not the Pod's service account. In my case, I was using GKE Autopilot that was configured via TerraForm. Because of a bug in the Terraform module, the service account used by the node is default (which is the default compute service account, e.g. <project_number>@cloudservices.gserviceaccount.com)

You can check which service account your nodepool is running with the gcloud container clusters describe ... command.

I hope this helps!