High Availability
Advanced 15 min read High Availability

HA Blueprints & Design Decisions

Choose your failures before they choose you

Putting the pieces together: reference architectures for 2- and 3-AZ (plus paired region) service tiers, with availability math, blast-radius planning, 'AZ-drain' drills, and the budget vs. nines tradeoff. Design is the practice of picking failures.

blueprintreference architectureblast radiusninesdrillfault treeWAF
Prerequisites: HA Database Patterns Multi-AZ Architecture
Before we start

What you will learn

  • Assemble the 3-AZ reference architecture for a tiered service
  • Trace availability math to a budget
  • Plan fault-injection drills + blameless design reviews
In 60 seconds

HA design = decide which single failure the system survives, then assemble redundancy (3-AZ compute, managed DB multi-AZ, queues/cache AZ-redundant) with 'the whole AZ can die' tolerances. Nines only come true after drills prove the wiring.

Introduction

What is this?

Reference: 3-AZ web+api (ALB), ASG/MIG/azure VMSS across zones, multi-AZ DB, SQS/S3, metrics RGW → dashboards. Regional: paired region for DR (see DR module), Global: route53+CloudFront. Availability budget: nines from parts (0.9999^components ≈), chain vs redundancy; realistically Nines ~= RTO/RPO enforcement not multiplication. Blast radius: batched deploys via lanes, feature flags, config rollout, data-destroying changes quarantined. Drills: AZ sim (stop whole-AZ instances in a test account), DB failover, DNS flip; game-day ≈ chaos engineering matured. Design review checklist (AWS Well-Architected HA pillar / Azure Reliability Hub): MTTR, fault domains, capacity slack vs peak, deviation from nines, and codes. Tradeoffs: double-infra cost, compliance (paired-region demands), ops overhead.

Motivation

Why do you need it?

  • The reference architecture answers 'what does decent look like' — you stop reinventing.
  • Availability math turns 'feels OK' into a defendable budget (and tells you where to stop paying).
  • Blast-radius design is where 'we're HA' meets 'how did THAT take prod down'.
  • Drills turn the blueprint from PowerPoint into muscle memory.
Think about it

Mental model

Imagine

A fire escape plan drilled quarterly: third floor → stair A, stairs blocked → stair B

Think of

HA drills

The blueprint (escape routes) is useless unless the fire team walks it (drill) enough to know stair B cold under stress. Same for AZ drains: rehearse until the flip is boring.

Mechanics

How it works

  • Define the lucky-failure: which single failure must be survived (AZ-loss? instance? DB node?).
  • Assemble 3-AZ compute + AZ-redundant data per the blueprint.
  • Set the budget: availability target distributes redundancy costs (which parts get 99.99%, which 99.9%).
  • Inject: quarterly AZ-drain, DB-failover, config-rollback-dance drills.
  • Blast-radius gates (lanes, flags, immutable image) plus review checklist run at design time.

3-AZ reference blueprint.

DNS/global
Ingress
Compute
Data
Ops
Hands-on

Examples

Availability budget sanity

Availability budget sanity
# target: service 99.95% (~ 263 min/yr)
# budget allocation:
#   LB/edge ...... .99999
#   compute fleet .9999
#   DB ............ .9999  (multi-AZ)
#   queues ........ .999
# crude: product ≈ .9995+... close to 99.95 with slack —
# but real practice: count WHAT fails, not percentages alone.

A crude chain; reality: MW-failures dominate. Pin the top failure modes per tier and spend the redundancy there.

Real world

In production

  • Adopt the 3-AZ blueprint for new services; retro-drill legacy via migration plan.
  • Quarterly game-day: kill an AZ's compute in a staging mirror, verify dashboards, time to health.
  • Well-Architected review on incident days to feed next-quarter HA goals.
Guidelines

Best practices

  • State the 'survivable failure' explicitly in an ADR; scope and agree on it.
  • Keep AZ-redundant data patterns managed (RDS/Aurora multi-AZ, S3) to reduce custom plumbing.
  • Drills are not optional: run failover, flap, and config rollback dance each quarter.
  • Size the slack: autoscalers + headroom must cover one-AZ loss within target RTO.
  • Treat deploys as HA events: lane-based, gated, revert-ready — never 'hot'.
Watch out

Common mistakes

  • Blueprint only vs. reality: 'we're 3-AZ' but any two pods have the same single instance volume backing them (zone fine, storage not).
  • Spending 99.999% effort on compute; the DB single-plane breaks the chain.
  • Drills skipped on 'the calendar is hard' → the surprise outage is the drill.
  • Cost-anxiety responses that chip away at redundancy until the 'HA' is a fetish.
When things break

Debugging tips

  • Drill shows failover OK but dashboards blank — dashboards/prom grabbed wrong zone; fix the monitor layer, not the app.
  • Capacity melts to 0 during quarter-hour — ASG max insufficient for one-AZ loss scenarios.
  • Two failures stacked during drill (AZ + DB flap) — understand chain: DB failover fan-out is your next drill.
  • Docs drying up post-drill — enforce drill reports feed the ADR/checklist to encode learnings.
Interviews

Interview checkpoint

Practice

Hands-on lab

Design + Drill a 3-AZ Blueprint
45 min · advanced
In progress
Objectives
  • On paper: assemble the tiers (LB/compute/data/queue) for a 92-99% target
  • Rehearse the AZ-drain motion against a local mirror
  • Write the ADR with survivable-failure + blast radius noted

Design + Drill a 3-AZ Blueprint

45 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

In HA design, 'blast radius' means:

2

The single syllable that makes blueprints trustworthy is:

Go deeper

Official documentation

AWS Well-Architected resilience pillar
docs
Azure reliability guidance
docs
GCP resilience best practices
docs