Alerting and On-Call for Data
Pages go to people with runbooks, not to email lists
Good alerting pages the right person, once, with a runbook attached. Design severity tiers, deduplicate, and keep on-call sustainable with clear ownership and escalation.
What you will learn
- Tier alerts by severity with explicit routing and escalation
- Deduplicate related failures into one incident
- Design sustainable on-call: ownership, handoff and postmortem culture
What is this?
Alerting converts metrics into actions. Good design has few rules: every alert has an owner, a severity, a threshold tuned to reality and an attached runbook; related alert noise is compressed into one incident; and on-call for data is a rota with boundaries (no 'everyone gets every page'), escalation and blameless follow-up. The goal is that when the pager fires, the person can say what broke, do first-response from the runbook and hand off cleanly.
Why do you need it?
- Alert storms burn out engineers and end with everything muted — including the important signals.
- A page without a runbook is a 2am research project; with one, it's a checklist.
- Expressive severity lets tier-1 triage sleep while tier-2 alerts page, and pages page p-worthy issues.
- Sustainable on-call keeps the loop closed: incidents become runbooks and better checks.
Mental model
A hospital triage nurse
A well-designed alert router
Not every symptom pages a surgeon. Triage sorts: red lights page owners immediately with a runbook; orange waits until business hours and routes to the right specialty; green goes to a dashboard. Alerting is the triage nurse for your data platform.
How it works
- Severity tiers: P1 (data outage, customers affected), P2 (degraded, delayed), P3 (single table/bug), P4 (info/trend).
- Routing: owner field on every metric drives the rota assignment; escalation to the owning team then their manager.
- Deduplication: group alerts by root cause (same run, same table) into one incident with all symptoms.
- Runbooks: each P1/P2 alert template includes verify-command, common causes, fix steps, escalation path.
- On-call rota: primary + secondary, defined ownership per domain, handoff notes, sleep-friendly P3 windows.
- Closed loop: after resolution, incident → postmortem → added regression check.
Alert lifecycle from metric to closed loop.
In production
- A payments platform routes P1 freshness-alerts to the owning squad's rota with a runbook that names the likely source connector and the verify SQL.
- A data team batches related nightly failures: when 12 tables go stale, on-call gets ONE incident with a checklist, not a dozen pages.
- An analytics org runs a P3-window (business-hours only) for non-critical drift, so nights stay quiet for genuinely critical issues.
Best practices
- Tune thresholds from baselines and review them with owners quarterly.
- Attach a runbook to every pageable alert and TEST the runbook on a real (or injected) incident.
- Deduplicate by root cause, keep consumers informed with one status, and auto-close on green.
- Never page without an owner; an alert with no owner is a wallpaper.
Common mistakes
- Paging everyone 'so somebody sees it' — the eventual response is everyone ignoring it.
- No runbooks → 1am tickets that take 90 minutes to even diagnose.
- Threshold churn: tweaking alert levels to silence noise instead of fixing the root causes.
- On-call with no handoff: secrets, caveats and semi-broken state passed verbally.
Debugging tips
- Page noise → split by severity and time-of-day; the truly noisy alerts often carry no runbook and no owner.
- Missed incident (should have paged, didn't) → the metric was sampled too slowly or the threshold was too high — inspect the monitor's burn history.
- Symptom clusters → group by common upstream run — one root often fans out into many alerts.
- Escalation dead-end → check the ownership map: alerts must route through the catalog's owner field, not hard-coded people.
Interview checkpoint
Hands-on lab
- Assign severity and owner from a metric
- Collapse related alerts into a single incident
- Escalate after a no-ack timeout
Route, Deduplicate and Escalate Alerts
30 min · intermediate
Quick quiz
The main reason 'page everyone' becomes ineffective:
Every pageable alert should have a runbook attached.