Lesson 8 of 8 in Observability and incident response
Blameless postmortems: learning after the incident
A postmortem is a written record of an incident: its impact, what was done to stop it, its causes and the follow-up work to prevent a repeat. A blameless postmortem looks for gaps in systems and processes instead of faulting a person, on the assumption that everyone acted reasonably with what they knew.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Incident response: triage, communicate, mitigate
The problem this solves
After an outage, the pull is to find who made the mistake. If people expect blame, they hide details, and the facts you need to prevent a repeat stay hidden. The same failure comes back with a different person at the keyboard.
Google's SRE book makes the point that you cannot fix people, but you can fix systems and processes. A question like why the deploy allowed a bad setting through is one a team can act on. A question like who deployed it is not.
A postmortem also fails when nothing happens afterwards. The value lies in follow-up actions with owners and dates, and in a document others can find and learn from.
How it works
- Other
- Runs code
- Control
- Data
After an incident you draft the postmortem, the team reviews it, the review produces owned action items, and the finished document is shared where others can find it.
- Incident ends to Draft: 1 write it up
- Draft to Review: 2 review together
- Review to Action items: 3 assign work
- Review to Shared library: 4 publish
Write the draft while the incident is fresh, using the timeline you kept. A good draft covers the impact (who was affected and for how long), what responders did, the root causes and contributing factors, and what the team will change.
Think in terms of contributing causes rather than one culprit. Most incidents involve several gaps, such as a missing check, a confusing alert and a rollout with no gradual stage. Each gap can be fixed.
The review is part of the process. Google's SRE book notes that an unreviewed postmortem might as well never have existed. After review, publish it in a place the whole team can search, so the lesson outlives the people who were on call.
Decide in advance what triggers a postmortem. The SRE book gives examples: user-visible downtime above a threshold, any data loss, an on-call intervention such as a rollback, or an incident found by hand because monitoring missed it.
Try it yourself
Write a postmortem for the lab incident in ~/obs-lab. Use the incident.md timeline from the previous concept. This is practice, so keep it short.
- 1Create the template
cd ~/obs-lab cat > postmortem.md <<'EOF' # Postmortem: <short title> ## Summary One or two sentences: what happened and for how long. ## Impact Who or what was affected, and how badly (use the error ratio you measured). ## Timeline (UTC) Paste from incident.md. ## Root causes and contributing factors Systems and processes only. No names. ## What went well ## What went badly ## Where we got lucky ## Action items | Action | Owner | Due | Type (prevent / detect / mitigate) | EOF
- 2Fill it in from your notes
cat incident.md
Copy the timeline into the template. For impact, quote the error ratio you saw. For causes, think about what let the bad setting reach users, for example no gradual rollout or no alert.
- 3Rewrite blame into systems
# practise on two sentences
Change 'I pushed a bad config' to 'a config change with a very high failure rate was applied with no validation step'. Change 'nobody noticed for ten minutes' to 'the alert required the condition to hold for a window that delayed notification, and no one was notified because no notifier was configured'. Notice that each rewrite points to something you can change.
- 4Write action items that can be finished
# example rows for the lab
A weak item is 'be more careful'. A strong item names an owner and a date, for example 'add Alertmanager so firing alerts reach a chat channel, owner: you, due: next week' or 'add a check that rejects an error rate setting above a limit'. Tag each one as prevent, detect or mitigate.
- 5Clean up the lab
cd ~/obs-lab docker compose down # when you no longer need the files: cd ~ && rm -rf ~/obs-lab
docker compose downstops and removes the containers and network. The folder holds only files from this topic. Remove it only when you have finished with them.
What breaks, and where to look
The postmortem names a person as the cause, and people start avoiding the on-call rotation or hiding details next time.
- WHY IT HAPPENS
- Blame teaches people to protect themselves rather than share facts.
- CHECK FIRST
- Reread each cause sentence. If it names a person, rewrite it as the missing check, unclear alert or gap in process that allowed the mistake.
The action items are vague, such as 'improve monitoring', and the same incident repeats.
- WHY IT HAPPENS
- An action without an owner, a date and a clear finish line does not get done.
- CHECK FIRST
- For each item ask who does it, by when, and how you will know it is done. Track the items in the same place as other work.
Postmortems are written but nobody reads them, and teams repeat old mistakes.
- WHY IT HAPPENS
- The documents were not reviewed or stored where others can find them.
- CHECK FIRST
- Hold a short review with the team, then save postmortems in one searchable place and link them from the services they concern.
When not to use this
- Not every blip needs a full document. Define triggers, such as user-visible downtime or data loss, and keep a short form for smaller events.
- Blameless does not mean consequence-free for ignoring clear policies repeatedly. It means that ordinary mistakes are treated as signs of a system gap.
Check yourself
What does 'blameless' mean in a postmortem?
It assumes people acted with good intent on the information they had, and looks for gaps in systems and processes rather than faulting individuals.
Name four things a postmortem should contain.
The impact, the actions taken to resolve it, the causes, and follow-up actions to prevent a repeat. A timeline is a usual part as well.
Why do action items need an owner and a date?
Without them nobody is responsible for finishing the work, and the same incident is likely to return.
Lesson 85 of 92 in the DevOps path