Incident Management & On-Call
The system under fire, run by the playbook
When it burns, respond like a team with a playbook: declare, communicate, mitigate-first, then learn with blameless postmortems — so the system improves every incident instead of repeating coasting through them.
What you will learn
- Structure an incident: roles, severity, comms, mitigation-first
- Write and rehearse runbooks
- Run a blameless postmortem that produces actions
Incident management is a practised routine, not a panic: an incident commander + scribe + responders, a severity ladder (before first customer-impacting SEV), mitigation-first (roll back, disable, redirect) over root-cause debate mid-fire, publiccomm comms on a cadence, and after: a blameless postmortem that finds system/process causes and ships action items with owners and dates.
What is this?
Detection → declaration → mitigation → resolution → postmortem. Detect via SLO burn alerts + user/on-call reports. Declare early (a 'declared incident, investigating' always beats silence) with a severity: SEV1 (customer-impacting, respond immediately), SEV2 (degraded, normal-hours), SEV3 (latent, tracked). Roles: commander (owns timeline + decisions, one brain decides), scribe (records everything, links trace IDs), responders (execute). Mitigation-first: the first move is usually rollback, feature-flag-off, traffic redirect, scale-out — restore service; root-cause can wait. Comms: a status channel/page updated every N minutes with 'what we know, what we're doing, next check-in'; external status page if applicable. Postmortem: blameless (the system failed, not the person), the timeline rebuilt from logs/pages, contributing causes + a causal chain, then action items with owners and deadlines — and the postmortem is only done when actions have dates and tracks them. Runbooks: pre-written, versioned, linked from alerts.
Why do you need it?
- A commander + scribe keeps the response coherent instead of a dozen people improvising in parallel.
- Mitigation-first minimizes user harm; the budget eats the diagnosis time, not the user.
- Blameless postmortems turn incidents into engineering investments, not blame careers.
- Runbooks make the common fire a 5-minute drill instead of a fresh detective hunt.
Mental model
A fire where everyone yells theory while it spreads
A fire drill: roles, hoses, and a timeline
The playbook works because the roles and steps existed before the smoke.
How it works
- Alerts (SLO burn) page the on-call; on-call declares + sets severity.
- Commander/scribe/responders form; comms cadence starts.
- Mitigate (rollback/flags/redirect); then investigate the cause safely.
- Resolve, then blameless postmortem with action items + owners.
The incident lifecycle.
In production
- A canary regressions pages on-call; commander flips the kill-flag in 4 minutes; resolution, cause investigation later.
- The 'deploy broke X again' incident produces a runbook addition, not a new human scapegoat.
- On-call handoff doc keeps the timeline, open actions, and next steps for the next shift.
Best practices
- Declare early, communicate on a cadence, fail-open the innocent.
- Mitigation first; cause analysis is a calm-hours activity.
- Every postmortem ships tracked action items.
- Runbooks are linked from the alerts that originate cases.
Common mistakes
- Debating root cause while users are down.
- Postmortems that list humans and vibe, not causes and actions.
- On-call without runbooks — the pager is a surprise test every time.
Debugging tips
- Two responders both patching prod → a commander will 'un-declare' stray changes with a revert.
- Postmortem with no action items → the next incident is already scheduled; force owners+dates.
Interview checkpoint
Quick quiz
First move in a SEV1?