Disaster Recovery
Intermediate 15 min read Disaster Recovery

Recovery Strategies: From Backup to Hot Standby

Same plan, faster — for a price

The recovery ladder: backup & restore (slowest), pilot light (skeleton), warm standby (scaled-down), multi-site hot standby, active-active (near-zero downtime). Each step trades cost for speed; pick per workload RTO.

pilot lightwarm standbyhot standbyactive-activeRTOrecoveryfailover
Prerequisites: Backup Strategies
Before we start

What you will learn

  • Lay out the recovery strategy ladder
  • Match a strategy to RTO/RPO and cost
  • Understand failover mechanics per tier
In 60 seconds

Backup-restore: hours–days, RPO hours unless PITR. Pilot light: core skeleton running (DB replica minimal, no app), minutes-to-1h. Warm standby: scaled-down but full stack, minutes. Hot standby: full size, near-zero promotion. Active-active: both serving, zero-downtime failover but hard (data sync, conflicts).

Introduction

What is this?

Tier mechanics: backup-restore (restore from snapshots, the ladder bottom — cost ~￥). Pilot light (provision only 'always-on' footprints: VPC, smallest DB, DNS entry; scale the rest on demand during DR). Warm standby (full stack smaller; upgrade on cutover), active-passive (standby at 50-100% with replication idle), active-active (multi-region traffic distribution + data conflict resolution; e.g., Aurora Global, CosmosDB multi-write, Spanner, CockroachDB w/ regional primary). Failover mechanisms: Route53 failover records/DNS flip, Traffic Manager priority, Cloud DNS steering; DB promote (RDS cross-region replica, promote-read), app image + IaC turn-up. Metrics: RTO cliff at each tier: B/R hours, pilot 30-60m, warm 15-30m, hot ~1m, AA near-zero.

Motivation

Why do you need it?

  • Cost is the enemy of DR — the ladder lets you spend exactly the 'speed' you need.
  • Matching strategy to RTO stops both overpaying (hot standby for a blog) and under-engineering (backup-restore for checkout).
  • Pilot light/warm standby give a vast RTO improvement for little spend, and are the sweet spot mid-size orgs need.
  • Each tier teaches failover mechanics that compose into the fanciest AA later.
Think about it

Mental model

Imagine

Fire station response: a volunteer crew on standby vs. professional staffed engines at the ready

Think of

Warm vs hot standby

Volunteers (pilot light) take time to rally a crew; trained staffed station (hot standby) rolls out in a minute. Which you fund depends on how fast the building must respond — same RTO-minded logic.

Mechanics

How it works

  • Choose tier per workload RTO/RPO (cheap where tolerable, hot only where painful).
  • Pilot: provision minimal skeleton in target region; DR = turn up the rest via IaC.
  • Warm: keep full stack smaller + replication live; DR = scale up + promote DB + flip DNS.
  • Hot: keep full-size standby; DR = promote/flip within a minute.
  • AA: both serving with data sync; DR = reroute traffic, absorb failover silently.

Recovery ladder.

Backup/restore
Pilot light
Warm standby
Hot standby
Active-active
Hands-on

Examples

Warm standby cutover sequence

Warm standby cutover sequence
# Warm standby: same stack, smaller, in region-B
# Cutover:
# 1. scale up app ASG/MIG (5m)
# 2. promote cross-region read replica → primary (2m, no data loss w/ sync option)
# 3. DNS flip via Route53 failover record (TTL-sync'd, ~1m)
# 4. smoke healthz + client shadow (2m)
# Total ≈ 10-15m — meets RTO 20m comfortably.

Steps are cheap-ish to run, and the whole cutover is rehearsable in a drill.

Real world

In production

  • Checkout/cart: warm or hot standby in a paired region.
  • Content/blogs: backup-restore + a doc'd 4h RTO (cost-conscious).
  • Core APIs: hot standby, cross-region replica, DNS flip, smoke test in the drill.
Guidelines

Best practices

  • Match strategy to the workload's real RTO, not the CEO's imprecise 'fast'.
  • Rehearse each tier's cutover exactly as production would run it.
  • IaC your turn-up so warm/pilot blue-sky is deterministic.
  • Keep the standby patched — a cold standby is a bypassed fire escape.
  • Define degradation expectations: during DR, some features may run reduced (accept that in the plan).
Watch out

Common mistakes

  • Funding 'active-active' for workloads that could use warm — block AHA for the bill.
  • Warm standby left unpatched/out-of-date images → cutover hits code drift.
  • DB promote without testing the sync → stale data surprises at go-live.
  • No DNS/edge cutover rehearsal: TTL secrets discovered in a real incident.
When things break

Debugging tips

  • Cutover slow at promote → measure the promote step; pre-warm replica + connection pool reset.
  • Traffic trickles to new region → client/DNS cache TTL or the LB security groups still closed.
  • Standby images old → CI should build + deploy to warm region every release (keeps it fresh).
  • AA conflicts during failover → your AP/write conflict resolution; test the merge paths pre-DR.
Interviews

Interview checkpoint

Practice

Hands-on lab

Walk a Warm-Standby Cutover
40 min · intermediate
In progress
Objectives
  • Model primary + standby regions with container 'regions'
  • Execute cutover: turn up app, promote DB, flip routing
  • Time it and compare to a target RTO

Walk a Warm-Standby Cutover

40 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Pilot light means:

2

The main cost lever in choosing a recovery tier:

Go deeper

Official documentation

AWS disaster recovery strategies
docs
Azure Site Recovery
docs
GCP DR options
docs