Grafana unified alerting shows NoData instead of firing or resolved
Grafana alert rule shows evaluation state 'NoData' instead of Normal or Alerting
Focused incident mode — strips away everything but this runbook.
Symptoms
- Grafana alert rule shows evaluation state 'NoData' instead of Normal or Alerting
- Alert history shows repeated transitions between NoData and Normal
- No notification is sent even though the underlying metric indicates an alert condition
- Grafana alert rule evaluation shows 'pending' indefinitely
Grafana NoData is Grafana's way of saying 'I cannot evaluate this right now.' It is a Grafana-side problem, not a Prometheus-side problem. Check Grafana's datasource health and alerting logs first.
Evidence to grab
Collect these artifacts before you change anything — they prove (or rule out) each cause.
Alert rule evaluation state
curl -s -H 'Authorization: Bearer <TOKEN>' http://<GRAFANA>/api/v1/provisioning/alert-rules | jq '.[] | select(.title == "<ALERT-TITLE>") | {uid, state, evalData}'Alert rule query and datasource
curl -s -H 'Authorization: Bearer <TOKEN>' http://<GRAFANA>/api/v1/provisioning/alert-rules/<RULE-UID> | jq '{query: .data, datasourceUid: .data[0].datasourceUid}'
Diagnostic flow
Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.
Guided decision tree is ready.
Steps with commands, expected output, and pass/fail branches.
Ranked root causes
Evidence checklist
- Datasource health check returns error or timeout
- Alert rule datasource UID points to a datasource that is down or misconfigured
- Grafana logs show 'context deadline exceeded' when evaluating the rule
Confirm before fixing
curl -s -H 'Authorization: Bearer <TOKEN>' http://<GRAFANA>/api/datasources/<DS-UID>/health | jq '.status'Check datasource connectivity
Fix
curl -s -X PUT -H 'Authorization: Bearer <TOKEN>' -H 'Content-Type: application/json' http://<GRAFANA>/api/datasources/<DS-ID> -d '{"url":"<CORRECT-PROMETHEUS-URL>","basicAuth":true,"basicAuthUser":"<USER>","secureJsonData":{"basicAuthPassword":"<PASS>"}}'Update the datasource URL and credentials
Resolution runbook
Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.
curl -s -H 'Authorization: Bearer <TOKEN>' http://<GRAFANA>/api/datasources/<DS-UID>/health | jq '.status'Check datasource health
curl -s -H 'Authorization: Bearer <TOKEN>' http://<GRAFANA>/api/v1/provisioning/alert-rules/<RULE-UID> | jq '{title: .title, evalFrequency: .evalFrequency, datasourceUid: .data[0].datasourceUid}'Inspect the alert rule configuration
curl -s -X PUT -H 'Authorization: Bearer <TOKEN>' -H 'Content-Type: application/json' http://<GRAFANA>/api/v1/provisioning/alert-rules/<RULE-UID> -d '{"evalFrequency":"60s"}'Increase evaluation interval if the query is slow
curl -s -X DELETE -H 'Authorization: Bearer <TOKEN>' http://<GRAFANA>/api/v1/provisioning/alert-rules/<RULE-UID>Delete the stale rule if it references a missing datasource or panel
Prevention
- Set evaluation intervals to at least 60s for slow queries; use recording rules for complex PromQL.
- Add a datasource health check panel and alert to every Grafana dashboard.
- Use alert rule provisioning (YAML files in version control) instead of UI-created rules.
- Monitor grafana_alerting_rule_evaluation_duration_seconds to detect slow evaluations.
Related runbooks
Same stack, same failure family — worth a skim.
Alert rule shows as 'inactive' or 'pending' but never reaches 'firing' state
Dashboard panels display 'No data' or an empty chart
Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.