highIntermediateUSE#prometheus#scrape#authentication#401#403

Prometheus scrape 401/403 — metrics endpoint requires authentication

Prometheus target shows last_error containing HTTP status 401 or 403

Likely source:configurationsecurity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Prometheus target shows last_error containing HTTP status 401 or 403
  • up == 0 despite the target being reachable and healthy
  • Scrape config has no authorization or basic_auth block
  • curl to the /metrics endpoint returns 'Unauthorized' without credentials

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Target last error

    curl -s http://<PROM-SERVER>:9090/api/v1/targets | jq '.data.activeTargets[] | select(.lastError != "") | {scrapeUrl, lastError}'

  • Scrape config for the target

    kubectl get cm -n monitoring prometheus-config -o yaml | grep -A20 '<TARGET-HOST>'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Target returns 401 without credentials
  • Prometheus scrape config has no auth block for this target
  • Target requires a bearer token or HTTP basic auth for /metrics

Confirm before fixing

kubectl exec -n monitoring deploy/prometheus -- wget -qO- http://<TARGET-HOST>:<PORT>/metrics 2>&1 | head -3
READ

Probe without credentials

Expected: HTTP/1.1 401 Unauthorized

Fix

kubectl edit cm -n monitoring prometheus-config
CHANGE

Add basic_auth or bearer_token to the scrape config for this target

Expected: ConfigMap updated

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl exec -n monitoring deploy/prometheus -- wget --server-response -qO- http://<TARGET-HOST>:<PORT>/metrics 2>&1 | head -5
READ

Confirm the 401/403 status

2
kubectl get cm -n monitoring prometheus-config -o yaml | grep -A20 '<TARGET-HOST>'
READ

Review the scrape config for this target

3
kubectl exec -n monitoring deploy/prometheus -- wget --header='Authorization: Bearer <TOKEN>' -qO- http://<TARGET-HOST>:<PORT>/metrics 2>&1 | head -3
READ

Test with valid credentials

4
kubectl create secret generic prometheus-scrape-creds -n monitoring --from-literal=bearer_token=<FRESH-TOKEN> --dry-run=client -o yaml | kubectl apply -f -
CHANGE

Rotate the token in Kubernetes

Prevention

  • Store scrape credentials in Kubernetes Secrets and reference them via bearer_token_file or basic_auth in scrape configs.
  • Set up a token rotation cronjob that updates the secret and triggers a Prometheus reload.
  • Use ServiceAccount token volumes (projected) instead of static tokens for long-lived scraping.
  • Alert on up == 0 per target for < 5 minutes to catch auth failures quickly.

Related runbooks

Same stack, same failure family — worth a skim.

Prometheus targets down — scrape failures with connection refused or DNS errorscritical

up == 0 for one or more targets on the /targets page

PrometheusFoundationUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.