mediumIntermediateSymptom-first#rbac#serviceaccount#forbidden#kubernetes

In-cluster RBAC Forbidden — pod can't call the Kubernetes API

Application logs show 'Forbidden: User cannot get resource pods'

Likely source:securityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Application logs show 'Forbidden: User cannot get resource pods'
  • Kubernetes client inside a pod gets 403 on the in-cluster API
  • Works with kubectl locally but not from the pod

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Which service account the pod runs under

    kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.serviceAccountName}'

  • Existing bindings

    kubectl get rolebinding,clusterrolebinding -n <namespace> | grep <sa-name>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • auth can-i shows 'no' for the required verbs
  • No binding exists between SA and a Role granting those verbs

Confirm before fixing

kubectl get rolebindings -n <namespace> -o json | jq '.items[].subjects // empty'
READ

Find bindings for the SA

Fix

kubectl create rolebinding <app>-read --clusterrole=view --serviceaccount=<namespace>:<sa-name> -n <namespace>
CHANGE

Bind the SA to an appropriate Role/ClusterRole

Expected: rolebinding.rbac.authorization.k8s.io/<app>-read created

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<sa-name>
READ

Confirm exactly which verbs the SA lacks

2
kubectl get rolebinding -n <namespace>
READ

Review existing bindings

3
kubectl create rolebinding <app>-read --clusterrole=view --serviceaccount=<namespace>:<sa-name> -n <namespace>
CHANGE

Grant the needed access (least privilege)

Prevention

  • One SA per workload with explicit Roles; no blanket cluster-admin.
  • Add RBAC self-review to CI (kubectl auth can-i in tests).
  • Use impersonation in console workflows instead of opening up permissions.

Related runbooks

Same stack, same failure family — worth a skim.

Long-lived secrets never rotate — leaked credentials go unnoticedhigh

A leaked API key keeps working for months

Security & SecretsAdvanced2 causes
Open runbook
RBAC Forbidden — serviceaccount cannot access API resourceshigh

kubectl logs show 'Error from server (Forbidden): pods is forbidden'

KubernetesIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.