Lesson 7 of 8 in Observability and incident response
Incident response: triage, communicate, mitigate
An incident is an event that disrupts a service for its users and needs a coordinated response. Incident response is the agreed way of detecting it, working out its impact, stopping the harm and communicating while it happens. The first aim is to restore service, and finding the root cause comes after.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Alerting and SLOs: paging on what users feel
The problem this solves
When a service breaks, the natural reaction is for everyone to look at it at once. People change things in parallel, nobody knows who did what, and users hear nothing. The outage lasts longer, and later nobody can say which action helped.
Teams that handle incidents well decide who leads, who changes the system and who talks to everyone else, and they do it before anything breaks. Google's SRE book describes these roles as incident command, operational work, communication and planning.
The other habit is to mitigate before you fully understand. If a change an hour ago caused the failure, undoing it restores service sooner than studying why it failed. The investigation can happen afterwards, calmly, with the full timeline.
How it works
- Control
- Runs code
- People and clients
The incident commander holds the overall picture and hands out work to an operations lead, a communications lead and a planning role, and communications keeps stakeholders informed.
- Incident commander to Ops lead: assigns fixes
- Incident commander to Communications: assigns updates
- Incident commander to Planning: assigns follow-up
- Communications to Stakeholders: regular updates
- Control
- Other
- Runs code
- Data
An alert starts the response, triage measures the impact, mitigation stops the harm, and a postmortem afterwards captures what to learn.
- Detect to Triage: 1 declare it
- Triage to Mitigate: 2 pick a safe fix
- Mitigate to Postmortem: 3 write it up
Start by declaring the incident. Google's SRE book suggests treating an event as an incident when a second team has to be involved, when customers can see the outage, or when it is still unsolved after an hour of focused analysis. Declaring early is cheap, and you can always close it quickly.
Triage answers three questions: what are users experiencing, how many are affected, and what changed recently. Metrics from the previous concepts give the first two. A list of recent deploys and config changes gives the third.
Mitigation means the least risky action that reduces harm: roll back a release, turn a feature off, add capacity, or send traffic elsewhere. The ops lead is the only person changing the system, so every change is known and recorded.
The commander can be one person doing several roles in a small team. What matters is that someone is clearly in charge, that changes go through one person, and that someone keeps others informed on a regular beat so they stop asking.
Try it yourself
Play a short incident on the lab app. You are every role at once. The bad change is a high failure rate, which simulates a faulty release.
- 1Open an incident notes file
cd ~/obs-lab cat > incident.md <<'EOF' # Incident notes Impact: Commander / Ops / Comms: ## Timeline (UTC) EOF echo "$(date -u +%H:%MZ) incident declared" >> incident.md
Write timeline entries as things happen. The shell adds the time for you.
- 2Cause the incident
ERROR_RATE=0.5 docker compose up -d app echo "$(date -u +%H:%MZ) simulated bad release deployed" >> incident.md
Give it a minute. In a real incident the first sign would be an alert, as in the previous concept.
- 3Triage: measure the impact
# paste into http://localhost:9090/graph sum(rate(app_requests_total{status="error"}[1m])) / sum(rate(app_requests_total[1m])) # then read what is failing docker compose logs --no-log-prefix --tail 10 appNote the error ratio and the log message in your notes. Latency looks normal, so this is failing requests, not slow ones.
- 4Mitigate first: roll back
ERROR_RATE=0.02 docker compose up -d app echo "$(date -u +%H:%MZ) rolled back to normal config, watching error ratio" >> incident.md
Undo the recent change before investigating why it was bad. Refresh the graph until the error ratio falls back.
- 5Close with a status line
echo "$(date -u +%H:%MZ) error ratio back to normal, incident resolved, postmortem to follow" >> incident.md cat incident.md
You now have a timeline to use in the next concept.
What breaks, and where to look
Several people are fixing at once, the error rate drops, and nobody knows which action helped.
- WHY IT HAPPENS
- There is no single person changing the system, so actions overlap and are not recorded.
- CHECK FIRST
- Name one ops lead, ask everyone else to suggest changes to that person, and log every change with its time in the shared notes.
An hour in, the team is deep in the code looking for the root cause while users still see errors.
- WHY IT HAPPENS
- The team is investigating before mitigating. Understanding takes longer than undoing a recent change.
- CHECK FIRST
- Ask what changed recently and whether it can be rolled back or switched off. Check your deploy history or
git logfor the last release.
Managers and support keep messaging engineers for status, which slows the fix.
- WHY IT HAPPENS
- Nobody has been assigned to communicate, so people ask the only source they know.
- CHECK FIRST
- Assign a communications role, post an update on a fixed beat such as every 30 minutes, and point everyone to that one place.
When not to use this
- A tiny hobby service does not need named roles. A written note, a rollback command and a habit of logging times cover the same ground.
- Do not declare a major incident for routine issues. Keep the process for events that affect users, or you will wear people out and lose the signal.
Check yourself
Why mitigate before finding the root cause?
Users are harmed while the service is broken. Undoing a recent change or turning off a feature usually restores service faster than diagnosing why it failed, and the investigation can follow.
Why should only one person change the system during an incident?
It keeps every change known and recorded, avoids conflicting actions, and lets you tell afterwards which action helped.
Name two signs that an event should be declared an incident.
Examples from Google's SRE book: a second team must be involved, customers can see the outage, or it is still unsolved after an hour of focused work.
Lesson 84 of 92 in the DevOps path