Lesson 8 of 8 in Observability and incident response

Blameless postmortems: learning after the incident

Core9 min

A postmortem is a written record of an incident: its impact, what was done to stop it, its causes and the follow-up work to prevent a repeat. A blameless postmortem looks for gaps in systems and processes instead of faulting a person, on the assumption that everyone acted reasonably with what they knew.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Incident response: triage, communicate, mitigate

The problem this solves

After an outage, the pull is to find who made the mistake. If people expect blame, they hide details, and the facts you need to prevent a repeat stay hidden. The same failure comes back with a different person at the keyboard.

Google's SRE book makes the point that you cannot fix people, but you can fix systems and processes. A question like why the deploy allowed a bad setting through is one a team can act on. A question like who deployed it is not.

A postmortem also fails when nothing happens afterwards. The value lies in follow-up actions with owners and dates, and in a document others can find and learn from.

How it works

From incident to lasting change
From incident to lasting changeAfter an incident you draft the postmortem, the team reviews it, the review produces owned action items, and the finished document is shared where others can find it.1 write itup2 reviewtogether3 assignwork4 publishIncident endsnotes and timelineDraftimpact, causes, timelineReviewthe team reads and asksAction itemsowner and due dateShared libraryothers can learn
  • Other
  • Runs code
  • Control
  • Data

After an incident you draft the postmortem, the team reviews it, the review produces owned action items, and the finished document is shared where others can find it.

  1. Incident ends to Draft: 1 write it up
  2. Draft to Review: 2 review together
  3. Review to Action items: 3 assign work
  4. Review to Shared library: 4 publish

Write the draft while the incident is fresh, using the timeline you kept. A good draft covers the impact (who was affected and for how long), what responders did, the root causes and contributing factors, and what the team will change.

Think in terms of contributing causes rather than one culprit. Most incidents involve several gaps, such as a missing check, a confusing alert and a rollout with no gradual stage. Each gap can be fixed.

The review is part of the process. Google's SRE book notes that an unreviewed postmortem might as well never have existed. After review, publish it in a place the whole team can search, so the lesson outlives the people who were on call.

Decide in advance what triggers a postmortem. The SRE book gives examples: user-visible downtime above a threshold, any data loss, an on-call intervention such as a rollback, or an incident found by hand because monitoring missed it.

Try it yourself

Write a postmortem for the lab incident in ~/obs-lab. Use the incident.md timeline from the previous concept. This is practice, so keep it short.

  1. 1
    Create the template
    cd ~/obs-lab
    cat > postmortem.md <<'EOF'
    # Postmortem: <short title>
    ## Summary
    One or two sentences: what happened and for how long.
    ## Impact
    Who or what was affected, and how badly (use the error ratio you measured).
    ## Timeline (UTC)
    Paste from incident.md.
    ## Root causes and contributing factors
    Systems and processes only. No names.
    ## What went well
    ## What went badly
    ## Where we got lucky
    ## Action items
    | Action | Owner | Due | Type (prevent / detect / mitigate) |
    EOF
  2. 2
    Fill it in from your notes
    cat incident.md

    Copy the timeline into the template. For impact, quote the error ratio you saw. For causes, think about what let the bad setting reach users, for example no gradual rollout or no alert.

  3. 3
    Rewrite blame into systems
    # practise on two sentences

    Change 'I pushed a bad config' to 'a config change with a very high failure rate was applied with no validation step'. Change 'nobody noticed for ten minutes' to 'the alert required the condition to hold for a window that delayed notification, and no one was notified because no notifier was configured'. Notice that each rewrite points to something you can change.

  4. 4
    Write action items that can be finished
    # example rows for the lab

    A weak item is 'be more careful'. A strong item names an owner and a date, for example 'add Alertmanager so firing alerts reach a chat channel, owner: you, due: next week' or 'add a check that rejects an error rate setting above a limit'. Tag each one as prevent, detect or mitigate.

  5. 5
    Clean up the lab
    cd ~/obs-lab
    docker compose down
    # when you no longer need the files:
    cd ~ && rm -rf ~/obs-lab

    docker compose down stops and removes the containers and network. The folder holds only files from this topic. Remove it only when you have finished with them.

What breaks, and where to look

  • The postmortem names a person as the cause, and people start avoiding the on-call rotation or hiding details next time.

    WHY IT HAPPENS
    Blame teaches people to protect themselves rather than share facts.
    CHECK FIRST
    Reread each cause sentence. If it names a person, rewrite it as the missing check, unclear alert or gap in process that allowed the mistake.
  • The action items are vague, such as 'improve monitoring', and the same incident repeats.

    WHY IT HAPPENS
    An action without an owner, a date and a clear finish line does not get done.
    CHECK FIRST
    For each item ask who does it, by when, and how you will know it is done. Track the items in the same place as other work.
  • Postmortems are written but nobody reads them, and teams repeat old mistakes.

    WHY IT HAPPENS
    The documents were not reviewed or stored where others can find them.
    CHECK FIRST
    Hold a short review with the team, then save postmortems in one searchable place and link them from the services they concern.

When not to use this

  • Not every blip needs a full document. Define triggers, such as user-visible downtime or data loss, and keep a short form for smaller events.
  • Blameless does not mean consequence-free for ignoring clear policies repeatedly. It means that ordinary mistakes are treated as signs of a system gap.

Check yourself

What does 'blameless' mean in a postmortem?

It assumes people acted with good intent on the information they had, and looks for gaps in systems and processes rather than faulting individuals.

Name four things a postmortem should contain.

The impact, the actions taken to resolve it, the causes, and follow-up actions to prevent a repeat. A timeline is a usual part as well.

Why do action items need an owner and a date?

Without them nobody is responsible for finishing the work, and the same incident is likely to return.

Lesson 85 of 92 in the DevOps path

Search

Search cheat sheets and projects