Lesson 7 of 8 in Observability and incident response

Incident response: triage, communicate, mitigate

Core13 min

An incident is an event that disrupts a service for its users and needs a coordinated response. Incident response is the agreed way of detecting it, working out its impact, stopping the harm and communicating while it happens. The first aim is to restore service, and finding the root cause comes after.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Alerting and SLOs: paging on what users feel

The problem this solves

When a service breaks, the natural reaction is for everyone to look at it at once. People change things in parallel, nobody knows who did what, and users hear nothing. The outage lasts longer, and later nobody can say which action helped.

Teams that handle incidents well decide who leads, who changes the system and who talks to everyone else, and they do it before anything breaks. Google's SRE book describes these roles as incident command, operational work, communication and planning.

The other habit is to mitigate before you fully understand. If a change an hour ago caused the failure, undoing it restores service sooner than studying why it failed. The investigation can happen afterwards, calmly, with the full timeline.

How it works

Who does what during an incident
Who does what during an incidentThe incident commander holds the overall picture and hands out work to an operations lead, a communications lead and a planning role, and communications keeps stakeholders informed.assigns fixesassigns updatesassigns follow-upregular updatesIncident commanderowns the big pictureOps leadonly one changing thingsCommunicationsupdates and statusPlanningnotes, bugs, handoffsStakeholdersusers, managers, teams
  • Control
  • Runs code
  • People and clients

The incident commander holds the overall picture and hands out work to an operations lead, a communications lead and a planning role, and communications keeps stakeholders informed.

  1. Incident commander to Ops lead: assigns fixes
  2. Incident commander to Communications: assigns updates
  3. Incident commander to Planning: assigns follow-up
  4. Communications to Stakeholders: regular updates
The order of work
The order of workAn alert starts the response, triage measures the impact, mitigation stops the harm, and a postmortem afterwards captures what to learn.1 declare it2 pick asafe fix3 write itupDetectalert or reportTriagewho and how badMitigatestop the harm firstPostmortemlearn afterwards
  • Control
  • Other
  • Runs code
  • Data

An alert starts the response, triage measures the impact, mitigation stops the harm, and a postmortem afterwards captures what to learn.

  1. Detect to Triage: 1 declare it
  2. Triage to Mitigate: 2 pick a safe fix
  3. Mitigate to Postmortem: 3 write it up

Start by declaring the incident. Google's SRE book suggests treating an event as an incident when a second team has to be involved, when customers can see the outage, or when it is still unsolved after an hour of focused analysis. Declaring early is cheap, and you can always close it quickly.

Triage answers three questions: what are users experiencing, how many are affected, and what changed recently. Metrics from the previous concepts give the first two. A list of recent deploys and config changes gives the third.

Mitigation means the least risky action that reduces harm: roll back a release, turn a feature off, add capacity, or send traffic elsewhere. The ops lead is the only person changing the system, so every change is known and recorded.

The commander can be one person doing several roles in a small team. What matters is that someone is clearly in charge, that changes go through one person, and that someone keeps others informed on a regular beat so they stop asking.

Try it yourself

Play a short incident on the lab app. You are every role at once. The bad change is a high failure rate, which simulates a faulty release.

  1. 1
    Open an incident notes file
    cd ~/obs-lab
    cat > incident.md <<'EOF'
    # Incident notes
    Impact:
    Commander / Ops / Comms:
    ## Timeline (UTC)
    EOF
    echo "$(date -u +%H:%MZ) incident declared" >> incident.md

    Write timeline entries as things happen. The shell adds the time for you.

  2. 2
    Cause the incident
    ERROR_RATE=0.5 docker compose up -d app
    echo "$(date -u +%H:%MZ) simulated bad release deployed" >> incident.md

    Give it a minute. In a real incident the first sign would be an alert, as in the previous concept.

  3. 3
    Triage: measure the impact
    # paste into http://localhost:9090/graph
    sum(rate(app_requests_total{status="error"}[1m])) / sum(rate(app_requests_total[1m]))
    # then read what is failing
    docker compose logs --no-log-prefix --tail 10 app

    Note the error ratio and the log message in your notes. Latency looks normal, so this is failing requests, not slow ones.

  4. 4
    Mitigate first: roll back
    ERROR_RATE=0.02 docker compose up -d app
    echo "$(date -u +%H:%MZ) rolled back to normal config, watching error ratio" >> incident.md

    Undo the recent change before investigating why it was bad. Refresh the graph until the error ratio falls back.

  5. 5
    Close with a status line
    echo "$(date -u +%H:%MZ) error ratio back to normal, incident resolved, postmortem to follow" >> incident.md
    cat incident.md

    You now have a timeline to use in the next concept.

What breaks, and where to look

  • Several people are fixing at once, the error rate drops, and nobody knows which action helped.

    WHY IT HAPPENS
    There is no single person changing the system, so actions overlap and are not recorded.
    CHECK FIRST
    Name one ops lead, ask everyone else to suggest changes to that person, and log every change with its time in the shared notes.
  • An hour in, the team is deep in the code looking for the root cause while users still see errors.

    WHY IT HAPPENS
    The team is investigating before mitigating. Understanding takes longer than undoing a recent change.
    CHECK FIRST
    Ask what changed recently and whether it can be rolled back or switched off. Check your deploy history or git log for the last release.
  • Managers and support keep messaging engineers for status, which slows the fix.

    WHY IT HAPPENS
    Nobody has been assigned to communicate, so people ask the only source they know.
    CHECK FIRST
    Assign a communications role, post an update on a fixed beat such as every 30 minutes, and point everyone to that one place.

When not to use this

  • A tiny hobby service does not need named roles. A written note, a rollback command and a habit of logging times cover the same ground.
  • Do not declare a major incident for routine issues. Keep the process for events that affect users, or you will wear people out and lose the signal.

Check yourself

Why mitigate before finding the root cause?

Users are harmed while the service is broken. Undoing a recent change or turning off a feature usually restores service faster than diagnosing why it failed, and the investigation can follow.

Why should only one person change the system during an incident?

It keeps every change known and recorded, avoids conflicting actions, and lets you tell afterwards which action helped.

Name two signs that an event should be declared an incident.

Examples from Google's SRE book: a second team must be involved, customers can see the outage, or it is still unsolved after an hour of focused work.

Lesson 84 of 92 in the DevOps path

Search

Search cheat sheets and projects