Monitoring
Intermediate 15 min read Monitoring

Alerting & Alert Fatigue

Alerting turns telemetry into pages that drive action. Learn alert rules, thresholds versus rates, severity, routing to PagerDuty and Slack, and how to keep alert fatigue from making your team deaf to real incidents.

alertingalert rulesalertmanagerseverityroutingPagerDutySlackon-callalert fatiguerunbook
Prerequisites: Observability Fundamentals Metrics & Prometheus
Introduction

What is this?

Alerting is the mechanism that turns telemetry into notifications someone must act on. In Prometheus, an alert rule runs a query on an evaluation interval and fires when a condition holds for a duration (the for clause). Firing alerts are grouped and routed by Alertmanager to receivers — PagerDuty pages, Slack messages, email or webhooks. Severity and routing decide who is paged and how loudly, while silences and good rule design keep alert fatigue low.

Motivation

Why do you need it?

  • Dashboards only help when someone is looking; alerts must reach a human even at 3am.
  • A good alert tells the on-call engineer what to do, not just what number is out of range.
  • Threshold-only alerts mislead; rate-of-change and SLO burn alerts catch trends before they break users.
  • Severity and routing ensure the right people are paged for the right problems, and nobody else.
  • Alert fatigue is the number one cause of missed incidents — every noisy rule trains your team to ignore the pager.
Think about it

Mental model

Imagine

A smoke detector

Think of

A good alert

A smoke detector tells you there is fire and where — that is the alert. Someone shouting 'something smells, maybe, not sure which room' is noise: it informs nobody and teaches everyone to ignore it. A good alert names the symptom, the impact and the next step; noise gets the whole system ignored.

Mechanics

How it works

  • An alert rule holds an expression, a condition like a threshold or a rate, and a for duration that the condition must persist.
  • Prometheus evaluates every rule on an evaluation interval (often 15-30s), moving the alert from inactive to pending when the condition is true.
  • When the condition has held for the for duration, the alert becomes firing and is pushed to Alertmanager.
  • Alertmanager groups related alerts by labels, applies routing to match a route, and delivers through a receiver such as PagerDuty or Slack.
  • Receivers page the on-call rotation; acknowledgements, escalation policies and silences govern what happens next.

The path from alert rule to a paged engineer.

Alert rule
Pending
Firing
Route
Receiver
Engineer
Reference

Key commands

curl -s http://localhost:9090/api/v1/rules

List all rules with their current evaluation state.

curl -s http://localhost:9090/api/v1/alerts

List alerts currently inactive, pending or firing.

curl -s http://localhost:9093/api/v2/alerts

Show the alerts Alertmanager currently tracks.

promtool check rules alerts.yml

Validate an alert rules file before loading it.

curl -X POST http://localhost:9090/-/reload

Hot-reload Prometheus configuration including rules.

Hands-on

Examples

An alert rule with a rate condition and a runbook

An alert rule with a rate condition and a runbook
groups:
  - name: checkout.rules
    rules:
      - alert: CheckoutErrorRateHigh
        expr: |
          (
            sum by (cluster) (
              rate(http_requests_total{job="checkout", status=~"5.."}[5m])
            )
            /
            sum by (cluster) (
              rate(http_requests_total{job="checkout"}[5m])
            )
          ) > 0.05
        for: 10m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout 5xx rate above 5% for 10 minutes"
          runbook: "https://runbooks.example.com/checkout-5xx"

This rule alerts on the rate of 5xx responses, not raw counts, so a slow day with many requests still fires correctly and a quiet night does not. The for: 10m clause waits out deploy blips, and the annotations carry the summary and a runbook link the on-call engineer can open instead of re-deriving the fix.

Real world

In production

  • The platform pages on SLO burn rate rather than every 5xx, so one flapping deploy produces one page, not forty.
  • Alerts are grouped by service and alert name so a multi-instance failure is a single incident, not a wall of messages.
  • Every firing alert's annotations include a runbook URL, cutting time-to-mitigation during incidents.
  • Maintenance windows are silenced at deploy time, so routine work does not page people.
  • A weekly alert review trims rules that never fired or that always fire, keeping the pager trustworthy.
Guidelines

Best practices

  • Alert on symptoms users feel — latency and errors — not implementation details like CPU usage on one host.
  • Use for: to wait out blips; page for minutes of a problem, not seconds of noise.
  • Make every alert actionable: a summary stating the impact and a runbook link in the annotations.
  • Set severity by customer impact, not by how worried the author is feeling today.
  • Alert on the rate of change (rate, increase, burn rate) rather than a raw gauge crossing a fixed number.
Watch out

Common mistakes

  • Paging on trivial thresholds that fire during every routine deploy.
  • Omitting for, so a single-sample blip pages someone at 3am.
  • Writing alert rules that merely duplicate a dashboard but add no decision.
  • Routing everything to one noisy channel so real pages get lost in chat.
  • Shipping alert rules that were never tested against historical data.
Root cause analysis

Common issues

Error
Alert fires then instantly resolves
Root cause

The rule has no for duration and reacts to a single scrape spike.

Fix

Add for: 5m and evaluate rate() over a window so transient blips do not page.

Error
Alert fires but nobody is notified
Root cause

The Alertmanager routing tree has no route matching the alert's labels.

Fix

Inspect the firing alert's labels and add a matching route with a configured receiver.

Error
The same incident produces dozens of duplicate alerts
Root cause

No grouping; each affected instance or series generates its own alert.

Fix

Configure group_by on the alert name or service labels so related alerts collapse into one notification.

When things break

Debugging tips

  • Test a rule before shipping it: promtool test rules replays it against real historical series.
  • Follow the state machine in /api/v1/alerts: inactive, then pending while for counts, then firing.
  • When a firing alert looks wrong, evaluate its exact expression manually in the UI to see the vector.
  • Check the route matchers in Alertmanager to explain why a page went to the wrong team.
  • Use a silence and verify it suppresses the alert to confirm routing before an incident.
Interviews

Interview checkpoint

Practice

Hands-on lab

Load an Alert Rule and Watch Its State
25 min · intermediate
In progress
Objectives
  • Write an alert rule with a for clause
  • Wire a rule file into a Prometheus container
  • Observe the alert state through the HTTP API
  • Verify the query that backs the rule

Load an Alert Rule and Watch Its State

25 min · intermediate

0/7 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which mechanism waits to confirm a condition persists before an alert fires?

2

Which alert summary best supports an on-call engineer?

3

More alert rules always mean fewer missed incidents.

Quick reference

Cheat sheet

Inspect Prometheus alerts
GET /api/v1/rulesRule evaluation states
GET /api/v1/alertsInactive, pending and firing alerts
promtool check rulesValidate rule files
POST /-/reloadApply config and rule changes
for: 10mRequire the condition to persist
Alertmanager
route matchersRoute alerts by label to teams
group_by: [alertname]Collapse duplicates into one notification
receiver: pagerdutyEscalation-grade notification
receiver: slackChat notification for non-urgent alerts
GET /api/v2/alertsCurrent alerts in Alertmanager
Go deeper

Official documentation

Prometheus — Alerting Overview
docs
Prometheus — Alerting Rules
docs
Prometheus — Alerting Best Practices
docs
PagerDuty Developer Documentation
docs