Backups, Disaster Recovery and Backfills
The warehouse died — now what, on paper, rehearsed?
DR for data isn't a backup job: it's RPO, RTO, multi-region copies and a restore-and-backfill runbook — designed, budgeted and rehearsed so 'the warehouse died' is an incident, not an extinction.
What you will learn
- Define RPO/RTO for your data platform per workload
- Design backups, multi-region copies and restorable states
- Plan and rehearse restore-and-backfill so recovery is a runbook, not a scramble
What is this?
Disaster recovery for data asks two numbers per workload: RPO — how much data you may lose (measured in time, e.g. 'the last 15 minutes'), and RTO — how long you have to recover. Then it designs the machinery behind them: object storage versioning/rebind loops, cross-region/cross-cloud copies, incremental backups, snapshots and replicas; plus a restore-and-backfill runbook that turns 'recovery' into a rehearsed sequence of restore → validate → reconcile → repoint consumers. DR that isn't tested is a wish.
Why do you need it?
- A single region or account can go away; without a DR design, your entire 'data platform' is one outage away from gone.
- RPO/RTO make recovery a budget conversation (how much money for how much safety), not a surprise.
- Rehearsed restores prove the backups actually work — most 'we have backups' orgs have never restored.
- Backfills are the routine sibling: replaying a window from raw/lineage is the same discipline as outage recovery.
Mental model
A bank vault with a rehearsed night-deposit recovery
Disaster recovery
The vault doesn't assume the branch never floods — it has copies offsite, a tested count-and-restore process, and a drill calendar. Data DR is the same: offsite copies, rehearsed restore, measured RTO.
How it works
- Define RPO/RTO per workload: e.g. core warehouse RPO=15min (cross-region copy), RTO=2h; archive RPO=1 day.
- Backups: daily DB snapshots + WAL/CDC-interval incremental; object versioning with lifecycle.
- Multi-region/cross-cloud: asynch replicas or periodic parquet copies of the analytic layer.
- Restore path: restore DB from snapshot+log replay; rebuild tables from raw/lineage where snapshots miss.
- Reconcile after restore: counts/keys/totals vs the pre-outage expectation (and raw source).
- Rehearse: a quarterly DR drill that restores a copy region and measures real RTO vs target.
DR design: copies, restore path, reconciliation and drill.
In production
- A warehouse snapshots nightly and ships CDC to a standby account every few minutes; a DR drill restores the standby to a 22-minute-old state in 1h40m across 40 tables.
- A lakes store versioned raw objects across two regions; a region outage recovers by re-materializing models from raw+lineage in 3 hours.
- An analytics org runs a scripted monthly backfill: replaying last month's window from raw via dbt, reconciling counts against baseline, and republishing.
Best practices
- Write RPO/RTO as numbers per workload on a published 'DR posture' sheet — budget follows the numbers.
- Backup the catalog/metastore too: restoring tables you can't find is restoring nothing.
- Test restore + reconcile, not just 'backup ran OK' — 'written' is not 'readable'.
- Automate backfills from lineage/raw so backlog replay is a command, not archaeology.
Common mistakes
- Backing up into the same region/account as production — the DR that dies with the primary.
- Never restoring until the real incident, then discovering 40% of backups are corrupt/outdated.
- RPO/RTO undefined, so recovery is 'whenever we get to it'.
- Backfill scripts that break because raw retention expired — the raw you planned to replay from is gone.
Debugging tips
- Restore fails but backups 'look fine' → test the restore job itself; most backup bugs are restore-side.
- Reconcile mismatch post-restore → identify which upstream window the snapshot missed; replay that partition from raw
- RTO met but RPO missed → incremental gap between snapshot and loss; shorten snapshot cadence or add log replay.
- Backfill drift → compare the replayed window's checksum/sum against raw; a line shuffle hides in aggregates.
Interview checkpoint
Quick quiz
Which is the most common overlooked DR failure?
Reconciling after a restore is as important as the restore itself.