criticalFoundationUSE#prometheus#scrape#targets#up-metric

Prometheus targets down — scrape failures with connection refused or DNS errors

up == 0 for one or more targets on the /targets page

Likely source:availabilityconnectivity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • up == 0 for one or more targets on the /targets page
  • last scrape error shows connection refused or dial tcp
  • Prometheus UI shows 'down' state in target list
  • Grafana panels relying on those targets go blank

up==0 is the single most common Prometheus alert. Treat it as a signal to investigate the target, not to restart Prometheus. Connection refused almost always means the exporter is dead inside the pod.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Target health status

    curl -s http://<PROM-SERVER>:9090/api/v1/targets | jq '.data.activeTargets[] | {scrapeUrl: .scrapeUrl, health: .health, lastError: .lastError}'

  • Scrape error in Prometheus logs

    kubectl logs -n monitoring deploy/prometheus | grep -i 'scrape error'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • lastError contains 'connection refused'
  • Target pod is Running but the exporter port is not bound
  • ss/netstat inside the container shows no listener on the metrics port

Confirm before fixing

kubectl exec -n <TARGET-NAMESPACE> deploy/<app> -- ss -tlnp | grep <PORT>
READ

Verify the exporter port is open

Expected: LISTEN 0.0.0.0:<PORT>

Fix

kubectl rollout restart deployment/<app> -n <TARGET-NAMESPACE>
CHANGE

Restart the target pod to bring the exporter back up

Expected: deployment.apps/<app> restarted

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -s http://<PROM-SERVER>:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "ok") | {scrapeUrl, health, lastError}'
READ

Identify which targets are down and why

2
kubectl exec -n monitoring deploy/prometheus -- wget -qO- --timeout=5 http://<TARGET-HOST>:<PORT>/metrics
READ

Probe the scrape endpoint from the Prometheus pod

3
kubectl get pod -n <TARGET-NAMESPACE> -l app=<app>
READ

Confirm the target pod is Running

4
kubectl rollout restart deployment/<app> -n <TARGET-NAMESPACE>
CHANGE

Restart the target if the exporter is dead inside a Running pod

Prevention

  • Add a readiness probe that checks the /metrics endpoint so Kubernetes removes non-scrapeable pods from Service endpoints.
  • Alert on up == 0 for more than 2 minutes with a per-target label.
  • Use pod labels (app, env) in scrape configs instead of raw DNS names when possible.
  • Monitor Prometheus scrape duration_seconds and samples_scraped to catch intermittent failures.

Related runbooks

Same stack, same failure family — worth a skim.

DNS resolution failing inside pods — CoreDNS issuescritical

curl: Could not resolve host inside pods

KubernetesIntermediate2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.