Lesson 7 of 7 in DevOps and delivery fundamentals

Blameless culture and learning from failure

Core8 min

A blameless postmortem is a written review of an incident that asks how the system made the failure possible, assuming people acted in good faith with the information they had. Its purpose is to find fixes to the system, not someone to punish.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: What DevOps is and why teams adopted it

The problem this solves

Every system fails eventually. What differs between teams is what happens next. If the first question is who did this, people learn to hide mistakes, report problems late, and avoid touching anything risky.

Hidden problems keep growing until they cause a bigger failure. Google's SRE book makes this point about postmortems: blame discourages people from raising issues, and people cannot be fixed, but systems and processes can be improved.

A blameless review keeps the facts and drops the verdict. It records what happened, why the actions made sense at the time, and what to change so the same slip is harder next time. Changing the system is the useful output.

How it works

From incident to a safer system
From incident to a safer systemAfter an incident the team builds a timeline, finds what made the failure possible, agrees on fixes, shares the write-up, and the system gains guardrails.1 whathappened2 why itmade sense3 fix list4 publish5 guardrails addedIncidentsomething brokeTimelinefacts, no verdictsContributing factorswhat made it possibleActionsowner and due dateShared write-upeveryone can read itGuardrailschecks, alerts, automation
  • Other
  • Control
  • People and clients
  • Data
  • Runs code

After an incident the team builds a timeline, finds what made the failure possible, agrees on fixes, shares the write-up, and the system gains guardrails.

  1. Incident to Timeline: 1 what happened
  2. Timeline to Contributing factors: 2 why it made sense
  3. Contributing factors to Actions: 3 fix list
  4. Actions to Shared write-up: 4 publish
  5. Shared write-up to Guardrails: 5 guardrails added

The postmortem begins with a timeline of events stated as facts. Then comes the central question: given what people knew at that moment, why did their actions seem reasonable, and what in the system allowed them to cause harm?

The answers become actions, each with an owner and a due date, tracked like any other work. The write-up is shared so other teams learn too. The real result, step 5, is a change to the system such as an automated check, an alert or a safer default.

Google's SRE book lists what a good postmortem contains: the impact, the steps taken to fix it, the root causes, and follow-up actions. Teams agree in advance on when to write one, for example for user-visible downtime, data loss, or a manual intervention.

Try it yourself

You will write a short postmortem about a fictional incident twice, once with blame and once without, and use a simple search to spot the difference. Linux, macOS or WSL.

  1. 1
    Write a draft that blames a person
    mkdir -p ~/devops-lab/blameless && cd ~/devops-lab/blameless
    cat > postmortem.md <<'EOF'
    # Example postmortem (fictional): checkout errors after a config change
    
    ## Summary
    The outage was the fault of the engineer who pushed the change. They should have checked the value first and were careless.
    
    ## Timeline
    - 14:02 config change merged
    - 14:05 checkout errors begin
    - 14:20 change reverted, errors stop
    
    ## Actions
    - Remind everyone to be more careful.
    EOF

    Everything here is made up for practice. Notice that the only action changes nothing about the system.

  2. 2
    Search for blame language
    grep -n -i -E "fault|should have|careless|more careful" postmortem.md

    Several lines match. This is a crude smell test, not a real review, but it shows how blame tends to sound.

  3. 3
    Rewrite it to describe the system
    cat > postmortem-fixed.md <<'EOF'
    # Example postmortem (fictional): checkout errors after a config change
    
    ## Summary
    A config change set a wrong value for the payment timeout. Nothing in the pipeline compared the new value with the allowed range, so it reached production.
    
    ## Timeline
    - 14:02 config change merged
    - 14:05 checkout errors begin
    - 14:20 change reverted, errors stop
    
    ## What made this possible
    - The review showed a diff of the file but not its effect on running services.
    - No automated check validates config values before merge.
    
    ## Actions
    - Add a config validation step to the pipeline (owner: platform team).
    - Roll config changes out to a small share of traffic first (owner: platform team).
    EOF
    grep -n -i -E "fault|should have|careless|more careful" postmortem-fixed.md
    echo "grep exit code: $?"

    You should see

    grep exit code: 1

    Exit code 1 means grep found no match. The second version names causes in the system and gives actions that change it.

What breaks, and where to look

  • A postmortem ends with human error as the root cause and an action to be more careful.

    WHY IT HAPPENS
    The review stopped at the first human action. People will make slips again, so an action that depends on perfect attention prevents nothing.
    CHECK FIRST
    Keep asking what made the slip easy and what would have caught it. Each action should change a system, such as a check, an alert, a default or an automation.
  • Problems reach the team only when users complain, and near-misses are never mentioned.

    WHY IT HAPPENS
    People fear criticism, so they hide or delay reports. Google's SRE book names this effect as the reason to avoid blame.
    CHECK FIRST
    Look at how the last incident was discussed. If anyone was singled out, expect fewer reports next time. Praise early reports of near-misses.
  • Postmortems are written, filed, and the same kinds of incident keep happening.

    WHY IT HAPPENS
    Actions have no owner or no priority, so they never compete successfully with feature work.
    CHECK FIRST
    Open the action items from the last three postmortems. Check each has an owner and a due date, and sits in the same backlog as other work.

When not to use this

  • Blameless does not mean nobody is accountable. It assumes good faith. Repeated disregard of agreed practice, or deliberate harm, is a management matter handled outside the postmortem.
  • Not every small blip needs a full write-up. Agree on triggers in advance, such as user-visible downtime, data loss or a manual intervention, and use a short note for the rest.

Check yourself

Why do teams avoid blame in a postmortem?

Blame makes people hide or delay reporting problems. Systems can be improved and people cannot be fixed, so the review looks for changes to the system.

What makes an action item useful?

It changes the system, such as adding a check or alert, and it has an owner and a due date.

Does blameless mean no accountability?

No. It assumes people acted in good faith with what they knew. Deliberate harm or ignoring agreed practice is handled separately.

Lesson 7 of 92 in the DevOps path

Search

Search cheat sheets and projects