Incident Response and Postmortems

• Run a declared incident with roles: commander, comms, responders
• Compress the timeline, mitigate first, investigate second
• Turn each incident into a postmortem with action items that narrow failure

• Untreated 'wait and see' costs hours; declared incidents get parallel effort and a comms loop.
• Roles remove the am-I-the-one problem; comms stop the 'is it just me?' chatter.
• Blameless postmortems get to the systemic truth instead of the scapegoat, which is what actually prevents recurrence.
• Action items owned with dates turn learning into change; a postmortem nobody acts on is a diary.

Nobody improvises a fire response at 3am — drill roles and exits are rehearsed. Data incidents get the same treatment: declared, role-driven, rehearsed plays, then a review that makes the next one shorter.

• Declare: severity call (P1/P2) as soon as an SLA risk is plausible; create the incident channel and page the rota.
• Roles: commander (decisions), comms (stakeholder updates), responder(s) (execution) — one person may wear two.
• Mitigate first: change to consistent state — rerun, failover, revert deploy, backfill — verification after stabilization.
• Timeline: record every action and timestamp (often auto-assembled from logs/traces).
• Postmortem: root cause, contributing factors, impact, corrective actions with owners and due dates.
• Follow-through: turn actions into tracked work; re-run exercises on the same scenario quarterly.

• A nightly warehouse run silently skips; on-call declares P1 after freshness breach, comms updates the newsletter, the squad reruns the load, and a postmortem adds a volume-vs-baseline check that catches it in 10 minutes next time.
• A streaming consumer falls behind for 6 hours; the postmortem's action item adds a lag SLO alert with a runbook — the next recur is a 20-minute downgrade, not a weekend.
• A data platform rehearses a quarterly scene (kill a broker, kill a backfill) and logs the exercise the same as a real incident.

• Declare early and downgrade later; declaring costs almost nothing and unlocks comms and parallel work.
• Mitigate before diagnosing. 'Why' is for the postmortem; 'restore' is the live job.
• Keep the timeline honest — postmortems are only useful if the record isn't laundered.
• Cap action items to the systemic top-3; 15 open items become zero closed items.

• Waiting for certainty before declaring — the 'maybe it self-heals' hour is pure cost.
• Fixing the root cause live instead of stabilizing first, so the outage stretches.
• A named-scapegoat postmortem that kills the next person's honesty.
• Action items without owners/dates, so learning evaporates by the next incident.

• Timeline first vs timeline second: build the timeline from logs/run manifests before theorizing.
• When mitigation options exist, pick the state-consistent one (revert vs patch) — patching live mid-incident creates new unknowns.
• Postmortem 'why' spirals → use five-whys up to system boundaries; stop at 'the system allowed it'.
• Track MTTD/MTTR trends per quarter; a shrinking MTTR is the leading indicator the loop works.