highIntermediateSymptom-first#iam#service account#key rotation#secret#credentials

Service account key rotation — broken key / compromised credential

Calls authenticate with an old service account key that now fails

Likely source:securityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Calls authenticate with an old service account key that now fails
  • Error: 'Invalid JWT Signature' or 'invalid_grant' from google-auth
  • A rotated/disabled SA key is still embedded in a config or CI pipeline
  • Security scan flags an exposed service account key

Key rotation is only complete when no client still holds the old JSON. Rotate in two phases: issue and distribute the new key, then disable and delete the old one after a grace window.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Key list and ages

    gcloud iam service-accounts keys list --iam-account=<SA_EMAIL> --managed-by=user

  • Which keys are enabled/disabled

    gcloud iam service-accounts keys list --iam-account=<SA_EMAIL> --managed-by=user --format='table(name,validAfterTime,validBeforeTime)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • The failing client's key id no longer appears in keys list
  • Key was rotated in IAM but the config still holds the old JSON
  • invalid_grant / Invalid JWT Signature errors

Confirm before fixing

gcloud iam service-accounts keys list --iam-account=<SA_EMAIL> --managed-by=user --format='value(name)' | grep -i <KEY_ID> || echo 'key not found'
READ

Confirm the old key id is gone

Expected: 'key not found' for the client's key id

Fix

gcloud iam service-accounts keys create <NEW_KEYFILE> --iam-account=<SA_EMAIL>
CHANGE

Create a fresh key and distribute it to clients

Expected: Created key [<KEY_ID>] for service account [<SA_EMAIL>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud iam service-accounts keys list --iam-account=<SA_EMAIL> --managed-by=user --format='table(name,validAfterTime,validBeforeTime)'
READ

Inventory keys and validity

2
gcloud iam service-accounts keys create <NEW_KEYFILE> --iam-account=<SA_EMAIL>
CHANGE

Create a replacement key for affected clients

3
gcloud iam service-accounts keys disable <OLD_KEY_ID> --iam-account=<SA_EMAIL>
CHANGE

Disable the old key after all clients migrate

Prevention

  • Rotate user-managed SA keys on a fixed schedule and enforce it with policys.
  • Prefer Workload Identity / metadata credentials over static JSON keys.
  • Store keys in Cloud Secret Manager and version them with the deploy.
  • Alert on invalid_grant and Invalid JWT Signature from your auth layer.

Related runbooks

Same stack, same failure family — worth a skim.

IAM policy propagation lag — newly granted role not effectivemedium

Role was just granted but calls still fail with Permission Denied

Google CloudFoundation3 causes
Open runbook
GKE Workload Identity not bound — GCE_SERVICE_ACCOUNT deniedhigh

Pod logs show permission denied when calling GCP APIs

Google CloudIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.