Disaster Recovery
Beginner 14 min read Disaster Recovery

Disaster Recovery Fundamentals

The plan for when the region itself dies

Disaster recovery is the plan for when your whole region goes away: RPO (how much data you can afford to lose) and RTO (how fast you must be back). Backups, replication, and a rehearsed recovery plan turn catastrophe into an operational event.

RTORPOdisaster recoveryrecovery planbackupreplicationDR
Prerequisites: HA Fundamentals
Before we start

What you will learn

  • Define RPO and RTO for a workload
  • Map DR strategies from backups to active-active
  • Understand the drill imperative
In 60 seconds

RTO = recovery time objective (how soon you must serve again). RPO = recovery point objective (how much data you lose). Strategies climb cost/effort: backups (chest, hours), pilot light (a skeleton, minutes), warm standby (scaled-down, minutes), multi-site active-passive/active-active (near-zero).

Introduction

What is this?

Definitions: RTO (time to service), RPO (data loss window), MTD (maximum tolerable downtime). Backup modes: snapshots (EBS/VM), managed backups, point-in-time recovery, cross-region copies. Replication: storage-level, DB binlog/WAL shipping, block-level. DR tiers (from DR panorama): backup & restore → pilot light → warm standby → hot standby (multi-site active-passive) → active-active (multi-region). Difference from HA: HA absorbs failures locally (instances/DB failover), DR handles region-level loss; the two compose — HA first, DR second. Cloud tooling: AWS Backup, Azure Backup, GCP Cloud Backup and DR, S3 cross-region replication, RDS cross-region replica, Cloud Formation disaster recovery runbooks.

Motivation

Why do you need it?

  • The region dying is not a myth — providers have had massive regional outages; plan for 'what if our whole stack is gone'.
  • RTO/RPO force the business to name what it can afford — before disaster decides for you.
  • A recovery plan + drill means 'our apps are back in ~target time' vs. 'let's figure it out'.
  • DR gives audit/compliance answers: regulators and customers ask about recovery plans.
Think about it

Mental model

Imagine

A ship's lifeboat drill: every crew member knows the boat, the route, and the muster point

Think of

DR plan

The lifeboat (backup) alone doesn't save you; knowing who rows, which route, and rehearsing the muster is what survives a real evacuation. Same with RTO/RPO + the drill.

Mechanics

How it works

  • Define RPO/RTO per workload (business impact analysis).
  • Choose the strategy that meets the numbers within budget.
  • Implement backups/replication with cross-region reach.
  • Write the recovery runbook: RACI, order of operations, verification steps.
  • Test the drill on a schedule and feed learnings back into the plan.

DR planning loop.

BIA
Pick strategy
Build/plumb
Drill
Learn
Hands-on

Examples

Workload DR profile

Workload DR profile
# e-commerce checkout
#   RTO: 2h   RPO: 15min
#   Strategy: warm standby (region-B scaled-down)
#     DB: cross-region replica, promote + cutover
#     App: golden image, autoscale to serve
#     Static: S3 CRR (near-instant)
#   Drill: quarterly; measured 1h50m last time

A profile ties budget to numbers: RPO 15m means near-continuous replication; RTO 2h rules out pure manual backup-restore.

Real world

In production

  • AWS Backup with cross-region copy policies + centralized vault.
  • RDS cross-region read replicas promoted during DR.
  • Pilot-light patterns: minimal stack in region-B, infrastructure-as-code turn-up.
Guidelines

Best practices

  • Document RTO/RPO per workload; revisit after every major change.
  • Rehearse the drill — 'tested in a lab' is not the same as 'drilled in Region B'.
  • Make DR infrastructure-as-code so the turn-up is deterministic and quick.
  • Cross-region backups: a backup in the dead region is a fantasy during the failure.
  • Define the decision signal: who declares DR, and what threshold (e.g., regional outage > 30 min).
Watch out

Common mistakes

  • Backing up into the same region that's dying (single point of loss).
  • RPO/RTO picked as aspirational numbers no plan actually meets.
  • DR plan never exercised → the runbook has gaps and stale IPs.
  • Treating HA as DR: failing over an instance is not recovering a region.
When things break

Debugging tips

  • Drill takes far longer than target → find the slow step (image creation, DNS, DB promote) and pre-stage it.
  • Restore-verified backups always seem fresh until you test one from tape/cold tier.
  • The app comes back but data is stale → RPO vs replication lag gap; measure yours.
  • Cross-region copy silently disabled after a storage change — audit the policy, not just the drill.
Interviews

Interview checkpoint

Practice

Hands-on lab

Write and Time a Basic Recovery Runbook
30 min · beginner
In progress
Objectives
  • Define RTO/RPO for one workload
  • Write a staged recovery runbook with RACI
  • Time a tabletop drill start-to-readiness

Write and Time a Basic Recovery Runbook

30 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

RPO describes:

2

The risk that undermines most DR plans:

Go deeper

Official documentation

AWS Disaster Recovery docs
docs
Azure disaster recovery
docs
GCP DR planning guide
docs