Production ETL Runbooks
The playbook you write before 3am
A runbook is the pipeline's 'how we get back to green' playbook: known failure signatures, step-by-step triage, rollback/redo plans, and post-incident notes. Write it before the incident and rehearse it.
What you will learn
- Structure a runbook: symptoms, triage, resolution, rollback
- Author known-issue signatures for your own pipelines
- Practice runbooks in drills, not just fire
What is this?
A runbook converts tribal knowledge into a searchable, executable playbook: symptoms (alert text), triage steps (which logs/checks), likely causes, fixes, and rollbacks. The discipline is writing one per pipeline, reviewing it quarterly, and noticing the incident where 40% of it was 'we already knew this'.
Why do you need it?
- On-call time is mostly spent re-discovering what the pipeline owner already knew.
- Rollback/replay decisions made late are what turn small issues into data incidents.
- A rehearsed runbook halves mean-time-to-recovery (MTTR) demonstrably.
Mental model
A pilot's emergency checklist vs improvisation
Runbooks
Under stress, pilots don't improvise — they run the checklist. The pipeline equivalent is a runbook that tells the on-call exactly which knobs to pull and what 'resolved' looks like.
How it works
- Symptom -> triage -> resolve -> verify: each section is a step with commands and expected outputs.
- Known signatures: 'rows_landed = 0 and source said 42' means watermark issue, not data loss.
- Rollback/redo explicitly: 'rerun with the same DAG and interval' versus 'revert schema first'.
- Post-incident: append lessons to the runbook so the next person inherits the learning.
- Drills: quarterly, break the pipeline on purpose and time the on-call against the runbook.
One incident through the runbook path.
Examples
A one-pager runbook entry
## PIPE: shop_daily | Owner: data-team SYMPTOM: "rows_landed = 0" at 03:15.TRIAGE: 1. Check source DB health (dashboard x-query). 2. airflow dags list-import-errors # parse? 3. Read last task attempts + logs.CAUSES (top): watermark stuck (no lookback), source pause, permission change.FIX: re-run idempotently after resolving cause; confirm rows > 0.ROLLBACK: none needed (raw landing immutable); re-run model leaf tasks.VERIFIED WHEN: published mart freshness < 4h and grain test passes.Specific, searchable, tested. The next on-call follows the same path the owner would.
In production
- A 40-pipeline org ships runbooks with each DAG; incidents drop from hours to minutes.
- A monthly drill takes the on-call rotation through three fake incidents against the runbooks.
- A post-incident review updates the runbook with the actual fix and its verification.
Best practices
- Every DAG gets a runbook with owner, alert, triage, fixes, rollback, and verify.
- Keep commands copy-pasteable and the verification step explicit.
- Review runbooks on a cadence; stale runbooks are as dangerous as missing ones.
- Email-tie drills to on-call onboarding so new members rehearse before needing to act.
Common mistakes
- One giant runbook nobody reads; everyone knows incidents will be improvised.
- Runbooks with only symptoms and causes — missing the 'exact command to fix it' step.
- Never updating the runbook after incidents, so the next failure hits version N-1.
Debugging tips
- When triage steps aren't followed, they were likely stale or wrong — fix the runbook, not the operator.
- Log every incident's actual fix; if it differs from the runbook, that's the review's first action item.
- Keep runbooks version-controlled next to the DAGs so they drift together or not at all.
Interview checkpoint
Quick quiz
A runbook should include the exact commands to fix or roll back AND a verification step.
The best time to write a pipeline's runbook is…