criticalIntermediateRED#prometheus#alerting#alertmanager#rules

Prometheus alert rule not firing — rule evaluated but alert stays inactive

Alert rule shows as 'inactive' or 'pending' but never reaches 'firing' state

Likely source:availabilityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Alert rule shows as 'inactive' or 'pending' but never reaches 'firing' state
  • Alertmanager receives no alerts for a condition that should be triggering
  • The PromQL expression returns data but the alert does not fire
  • Grafana shows the alert rule but the alert history is empty

An alert that never fires is worse than no alert at all. Always validate new rules with promtool test rules and fire a synthetic alert via the Alertmanager API before relying on them.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Alert rule state

    curl -s http://<PROM-SERVER>:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name == "<ALERT-NAME>") | {state, lastError, activeAt}'

  • Alertmanager connection status

    curl -s http://<PROM-SERVER>:9090/api/v1/alertmanagers | jq '.data'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • alert rule state=err or lastError contains 'undefined variable'
  • The query returns empty even though the condition should be true
  • The metric was renamed or removed in a recent exporter update

Confirm before fixing

curl -s 'http://<PROM-SERVER>:9090/api/v1/query?query=<ALERT-EXPR>' | jq '.data.result'
READ

Run the alert expression as an instant query

Expected: Empty array or error

Fix

promtool check rules <RULE-FILE-PATH>
READ

Validate the rule file and fix the expression

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
curl -s http://<PROM-SERVER>:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name == "<ALERT-NAME>") | {state, health, lastError}'
READ

Check alert rule state and health

2
curl -s 'http://<PROM-SERVER>:9090/api/v1/query?query=<ALERT-EXPR>' | jq '.data.result'
READ

Test the alert expression directly

3
curl -s http://<PROM-SERVER>:9090/api/v1/alertmanagers | jq '.data.activeAlertmanagers'
READ

Verify Alertmanager connectivity

4
curl -s -X POST http://<PROM-SERVER>:9090/-/reload
CHANGE

Reload config if the rule was recently changed

Prevention

  • Run promtool test rules in CI for every new or modified alert rule.
  • Alert on Prometheus rule evaluation errors: Prometheus evaluating rules group interval metric.
  • Monitor the alertmanager_notifications_total metric to confirm alerts are being sent.
  • Use a dead-man's-switch alert (always firing) to verify the Prometheus → Alertmanager pipeline end-to-end.

Related runbooks

Same stack, same failure family — worth a skim.

Prometheus rules file fails config reload with parse errorhigh

Prometheus config reload fails with 'error parsing YAML' or 'parse error'

PrometheusIntermediate3 causes
Open runbook
Grafana dashboard panel shows 'Datasource with UID ... not found' after migrationhigh

Dashboard panels display 'Datasource with UID <uid> not found'

GrafanaIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.