highAdvancedSymptom-first#secrets#rotation#credentials#crypto

Long-lived secrets never rotate — leaked credentials go unnoticed

A leaked API key keeps working for months

Likely source:security

Focused incident mode — strips away everything but this runbook.

Symptoms

  • A leaked API key keeps working for months
  • No rotation policy or automation exists for a database / cloud credential
  • Rotation breaks consumers because the new value is not propagated

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Credential age

    aws secretsmanager describe-secret --secret-id <SECRET> | grep -E 'LastRotatedDate|RotationEnabled'

  • Component consuming the secret

    grep -r '<SECRET_KEY>' --include='*.env' --include='*.yaml' --include='*.ts' config/ 2>/dev/null

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • RotationEnabled false on the manager entry
  • Credential committed into the repo years ago with no expiry

Confirm before fixing

aws secretsmanager describe-secret --secret-id <SECRET> --region <REGION> | grep RotationEnabled
READ

Confirm rotation disabled

Fix

aws secretsmanager rotate-secret --secret-id <SECRET> --rotation-lambda-arn <LAMBDA> --rotation-rules AutomaticallyAfterDays=30
CHANGE

Enable Lambda-based rotation

Expected: Rotation scheduled every 30 days

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws secretsmanager describe-secret --secret-id <SECRET> --region <REGION> | grep -E 'RotationEnabled|LastRotatedDate'
READ

Assess current rotation state

2
aws secretsmanager rotate-secret --secret-id <SECRET> --rotation-lambda-arn <LAMBDA> --rotation-rules AutomaticallyAfterDays=30
CHANGE

Enable rotation

3
kubectl rollout restart deployment -A -l app=<app>
CHANGE

Re-deploy consumers to pick up the rotated value

Prevention

  • Enable KMS/Secrets-Manager rotation on day one; rotate on schedule.
  • Never embed secrets in git — use injected references (IAM roles, Vault plan, sealed secrets).
  • Alert on leaked-credential findings (Secret Scanning / GuardDuty) with an SLA to rotate in 24h.

Related runbooks

Same stack, same failure family — worth a skim.

In-cluster RBAC Forbidden — pod can't call the Kubernetes APImedium

Application logs show 'Forbidden: User cannot get resource pods'

Security & SecretsIntermediate2 causes
Open runbook
Security group / firewall open to the world — exposed servicecritical

Port 22 or 5432 reachable from any public IP

Security & SecretsFoundation2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.