mediumFoundationSymptom-first#iam#propagation#role#binding#delay

IAM policy propagation lag — newly granted role not effective

Role was just granted but calls still fail with Permission Denied

Likely source:configurationsecurity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Role was just granted but calls still fail with Permission Denied
  • gcloud iam policy shows the binding but the API rejects it
  • A new member cannot access a resource for a few minutes after grant
  • Same permission works for other members, only the new one is slow

IAM changes propagate within minutes; a role that works for other members but not a newly added one is usually propagation, not a missing binding. Do not thrash the policy — verify it once and wait.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Policy state as seen by the policy API

    gcloud projects get-iam-policy <PROJECT> --flatten='bindings[].members' --filter='bindings.members=<MEMBER>' --format='value(bindings.role)'

  • Cached credential / token time

    gcloud auth print-access-token | cut -d. -f2 | base64 -d 2>/dev/null | head -c 200

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Binding visible in policy but resource denies
  • Succeeds after a few minutes without further changes
  • Consistent with GCP's IAM eventual-consistency window

Confirm before fixing

gcloud projects get-iam-policy <PROJECT> --flatten='bindings[].members' --filter='bindings.members=<MEMBER>' --format='value(bindings.role)'
READ

Confirm the binding is registered

Expected: Role present

Fix

gcloud projects add-iam-policy-binding <PROJECT> --role=<ROLE> --member=<MEMBER>
CHANGE

Re-assert the binding (idempotent) while propagation settles

Expected: Updated IAM policy for project [<PROJECT>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud projects get-iam-policy <PROJECT> --flatten='bindings[].members' --filter='bindings.members=<MEMBER>' --format='value(bindings.role)'
READ

Confirm the grant is registered

2
gcloud auth print-access-token
READ

Check whether the token predates the change

3
gcloud auth login --force
CHANGE

Refresh credentials if the token is stale

4
curl -s -o /dev/null -w '%{http_code}' '<RESOURCE_URL>'
READ

Retest the originally failing call

Prevention

  • Wait a short propagation window in automated provisioning before asserting function.
  • Use precondition checks (test the permission, not the policy) in canaries.
  • Version role grants with the deploy that needs them.
  • Alert on permission-denied in the first minutes after a known role rollout.

Related runbooks

Same stack, same failure family — worth a skim.

Service account key rotation — broken key / compromised credentialhigh

Calls authenticate with an old service account key that now fails

Google CloudIntermediate3 causes
Open runbook
GKE Workload Identity not bound — GCE_SERVICE_ACCOUNT deniedhigh

Pod logs show permission denied when calling GCP APIs

Google CloudIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.