Disaster Recovery
Intermediate 13 min read Disaster Recovery

DR Drills & Testing

Plans are fiction until rehearsed

The plan is only as real as the drill: tabletop tests, full fallover drills, restore tests, chaos-game days. Frequency, scope, measure-against-RTO, blameless after-action — this is where DR becomes muscle memory.

drillgame daytabletopfallover testrestore testafter-action
Prerequisites: DR Fundamentals
Before we start

What you will learn

  • Design a drill ladder: tabletop → business continuity → full outage sim
  • Measure drills against RTO/RPO targets
  • Run blameless after-action with actionable items
In 60 seconds

Drill types: tabletop (walk the plan), component tests (restore one workload), business continuity (partial outage), full game day (whole companion copy degraded). Success = whatever the drill reveals + measured progress vs RTO; failures are findings, not blame.

Introduction

What is this?

Drill ladder: 1) tabletop (runbook walk-through, no infra) monthly; 2) restore-test individual workloads (DB/upload) quarterly; 3) failover drill of one service + DNS (quarterly); 4) full-site game day (twice a year ideally). Measurement: observe RTO/RPO metrics, count of runbook deviations, steps that failed, time each stage. Tooling: AWS Fault Injection Simulator, chaos tools (Chaos Mesh, Gremlin), provider 'disaster' simulations (Azure, GCP test failovers), DR runbook repositories with checklists. After-action: blameless retro: timeline, impact, what went well, top 3 findings with owners + deadlines; wire actionable into docs/ADR/IaC. Culture: schedule + calendar protection; leadership treats drills as feature work; findings tracked like bugs. Pitfall: drills in a 'perfect' test env that never mirror prod quirks.

Motivation

Why do you need it?

  • Orchestration only reveals gaps when exercised under pressure.
  • Drills convert the runbook from prose to practiced behaviors — the difference between 'we planned' and 'we recovered'.
  • Restore/down tests catch the boring-machine failures (permissions, stale images, DNS TTL) before they're 3am bugs.
  • Measured RTO across drills trains the whole org's time sense.
Think about it

Mental model

Imagine

Fire drills: first at school, then exit drills in offices; repetition makes the route instinctive

Think of

DR drills

Nobody learns the fire route by reading it; they walk it. Same with DR: the fourth walk beats the first 'plan'. And each walk reveals a door that was locked.

Mechanics

How it works

  • Schedule a ladder of drills (tabletop to game day) with owners and dates committed.
  • Each drill: define scope, trigger, success = meeting RTO/RPO in the cleanest path.
  • Time stages; capture deviations (runbook wrong, tooling missing, DNS wrong TTL).
  • After-action: blameless retro within 48h; top findings with owners.
  • Findings become tracked actions → plan updated → next drill retests them.

Drill loop.

Schedule
Trigger
Measure
Retro
Improve
Hands-on

Examples

Game day scorecard

Game day scorecard
# Quarterly full-site game day
# Target RTO 20m / RPO 15m
# Stage timings:
#   T+0   declare + page          (2m)
#   T+2   IaC apply to region-B    (8m)
#   T+9   DB promote               (2m, RPO 22s lag measured)
#   T+11  DNS flip                 (2m)
#   T+13  smoke + handoff          (4m)
#   TOTAL 17m — green vs RTO 20m.
# Findings: 2 (1 doc fix, 1 standby image 2-missed).  Owner set.

Numbers make the review: the drill measured 17m, beat target, and surfaced two biteable findings.

Real world

In production

  • Quarterly restore-test on the crown-jewel DB (restore to staging, row-count verify).
  • Bi-annual full game day: run DR against region-B copy; measure every stage.
  • After-actions wired to the tracker and the ADR — drill findings are feature-queue input.
Guidelines

Best practices

  • Drill the stair-step: tabletop → single-workload → service → site — scale up, don't boil the ocean.
  • Choose a 'production-ish' mirror; a clean lab is a nicer fiction, not a truer one.
  • One clear success metric (met RTO, key flows healthy) and time per stage recorded.
  • Never let blame leak into retros; 'who' is a symptom, 'what/why' is the finding.
  • Calendar commitments: drills are feature work — reschedule like an incident, not a nicety.
Watch out

Common mistakes

  • Tabletop only, forever: the plan gets walked, never stood up.
  • Game day in a spotless lab with prod's quirks engineered out.
  • Measuring 'we did it' instead of 'we did it within RTO at X lag'.
  • After-actions evaporate: findings without owners die on the retro wall.
When things break

Debugging tips

  • A drill keeps missing RTO at the same stage → deep-dive that stage (pre-stage infra, images, permissions) before the next drill.
  • Restore a month of data but checksums drift → the backup pipeline is skipping tables/transactions; inspect copy logs.
  • Post-drill findings look intentional → insufficient blameless-ness; reinforce culture signals.
  • Drill cadence slides → schedule them as recurring change windows with an RTO-violating business-impact narrative.
Interviews

Interview checkpoint

Practice

Hands-on lab

Tabletop + One-Service DR Drill
30 min · intermediate
In progress
Objectives
  • Run a 30-min tabletop on a chosen workload
  • Execute a real one-service restore failover
  • Write the after-action findings

Tabletop + One-Service DR Drill

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The strongest signal that a DR plan is real:

2

After-action rule that keeps retros effective:

Go deeper

Official documentation

AWS fault-tolerant drills
docs
Azure SRE drills
docs
Google's Game Day handbook
docs