highIntermediateSymptom-first#gke#workload identity#iam#service account#gce_service_account

GKE Workload Identity not bound — GCE_SERVICE_ACCOUNT denied

Pod logs show permission denied when calling GCP APIs

Likely source:configurationsecurity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Pod logs show permission denied when calling GCP APIs
  • Error mentions GCE_SERVICE_ACCOUNT or 'the default service account' with access denied
  • gcloud/GCS client inside the pod fails with 403 despite valid ADC
  • Newly enabled Workload Identity pod still uses the node's default SA

Workload Identity needs three things in place: pool metadata mode, namespace+KSA annotations, and the IAM workloadIdentityUser binding to the KSA member. Missing any one falls back to the shared node SA.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Pod's projected token and used identity

    kubectl exec <POD> -n <NS> -- sh -c 'ls /var/run/secrets/tokens && cat /var/run/secrets/iam.googleapis.com/serviceaccount/email 2>/dev/null'

  • Namespace annotation

    kubectl get namespace <NS> -o jsonpath='{.metadata.annotations}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • No annotation on namespace or KSA
  • Error logs show the node's default GCE_SERVICE_ACCOUNT being used
  • Workload identity metadata mode disabled on the pool

Confirm before fixing

kubectl get namespace <NS> -o jsonpath='{.metadata.annotations.iam.gke.io/gcp-service-account}'
READ

Confirm the annotation is present

Expected: Empty output when the annotation is missing

Fix

kubectl annotate namespace <NS> iam.gke.io/gcp-service-account=<GSA_EMAIL> --overwrite
CHANGE

Bind the namespace to the GCP service account

Expected: namespace/<NS> annotated

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl annotate namespace <NS> iam.gke.io/gcp-service-account=<GSA_EMAIL> --overwrite
CHANGE

Bind the namespace to the IAM SA

2
kubectl annotate serviceaccount <KSA> -n <NS> iam.gke.io/gcp-service-account=<GSA_EMAIL> --overwrite
CHANGE

Bind the KSA to the IAM SA

3
gcloud iam service-accounts add-iam-policy-binding <GSA_EMAIL> --role=roles/iam.workloadIdentityUser --member='serviceAccount:<PROJECT>.svc.id.goog[<NS>/<KSA>]'
CHANGE

Grant workloadIdentityUser to the KSA

4
kubectl rollout restart deployment/<DEPLOY> -n <NS>
CHANGE

Redeploy pods so they pick up the new identity

Prevention

  • Bind namespace+KSA and grant IAM role in the same change (idempotent IaC).
  • Enable GKE_METADATA_SERVER on every pool at creation time.
  • Add a test pod that validates the resolved identity (gcloud auth list inside pod).
  • Do not grant broad roles to the node's default SA — that defeats WI's purpose.

Related runbooks

Same stack, same failure family — worth a skim.

GKE unable to pull image from Artifact Registry (permissions)critical

Pods stuck in ImagePullBackOff or ErrImagePull

Google CloudIntermediate3 causes
Open runbook
IAM policy propagation lag — newly granted role not effectivemedium

Role was just granted but calls still fail with Permission Denied

Google CloudFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.