criticalIntermediateSymptom-first#gke#artifact registry#iam#permissions#image pull

GKE unable to pull image from Artifact Registry (permissions)

Pods stuck in ImagePullBackOff or ErrImagePull

Likely source:configurationavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Pods stuck in ImagePullBackOff or ErrImagePull
  • Events show 'denied: Permission "artifactregistry.repositories.downloadArtifacts" denied'
  • kubectl describe pod shows Failed to pull image 'gcr.io/<PROJECT>/<IMAGE>'
  • Image pulls work locally with gcloud but fail inside the cluster

GKE pulls images using the node pool's service account, not the pod's Workload Identity. Grant artifactregistry.reader to the pool SA (or use pull secrets) to fix pull denials.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Pull error in pod events

    kubectl describe pod <POD> -n <NS> | grep -A5 Events

  • Node pool SA used for pulls

    gcloud container node-pools describe <POOL> --cluster <CLUSTER> --zone <ZONE> --format='value(config.serviceAccount)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Event message names artifactregistry.repositories.downloadArtifacts denied
  • Repo is private while the SA has no roles/artifactregistry.reader
  • Token from the node SA fails to pull in a container

Confirm before fixing

gcloud projects get-iam-policy <PROJECT> --flatten='bindings[].members' --format='table(bindings.role,bindings.members)' | grep -i artifactregistry
READ

Check Artifact Registry bindings in the project

Expected: roles/artifactregistry.reader bound to the node SA (or a group containing it)

Fix

gcloud projects add-iam-policy-binding <PROJECT> --role=roles/artifactregistry.reader --member=serviceAccount:<NODE_SA>
CHANGE

Grant the node SA artifactregistry.reader

Expected: Updated IAM policy for project [<PROJECT>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl describe pod <POD> -n <NS> | grep -A6 'Events:'
READ

Confirm the exact pull denial

2
gcloud projects add-iam-policy-binding <PROJECT> --role=roles/artifactregistry.reader --member=serviceAccount:<NODE_SA>
CHANGE

Grant the node SA reader access

3
gcloud artifacts repositories list --format='table(name,location,format)'
READ

Verify the repo path is correct

4
kubectl rollout restart deployment/<DEPLOY> -n <NS>
CHANGE

Retrigger the image pull after fixing perms

Prevention

  • Grant roles/artifactregistry.reader to node pools at creation via IaC.
  • Use regional image paths matched to your repo location to avoid 404s.
  • Monitor ImagePullBackOff events as a service-health alert.
  • Pin image digests for reproducible, cache-friendly pulls.

Related runbooks

Same stack, same failure family — worth a skim.

ImagePullBackOff / ErrImagePull — pod can't fetch its imagehigh

Pod stuck in ImagePullBackOff or ErrImagePull

KubernetesFoundation3 causes
Open runbook
GKE Workload Identity not bound — GCE_SERVICE_ACCOUNT deniedhigh

Pod logs show permission denied when calling GCP APIs

Google CloudIntermediate3 causes
Open runbook
IAM policy propagation lag — newly granted role not effectivemedium

Role was just granted but calls still fail with Permission Denied

Google CloudFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.