High Availability
Advanced 16 min read High Availability

Highly Available Database Patterns

Every nines drama has a database at its heart

The database is the HA crown jewel: sync replication with auto-failover (managed multi-AZ), read replicas for load, and quorum-based avoids of split-brain. Understand RPO/RTO semantics of each pattern and test failover regularly.

multi-AZreplicationfailoverquorumsplit-brainRPOread replicaMongo
Prerequisites: Managed Databases Multi-AZ Architecture
Before we start

What you will learn

  • Compare managed multi-AZ vs custom replication
  • Design read-scaling with replicas
  • Explain quorum/split-brain avoidance
In 60 seconds

Managed multi-AZ: primary + sync standby in another AZ, failover flips DNS (RDS ~60s, Azure SQL seconds, Cloud SQL ~60s) — RPO=0 (no sync loss), RTO seconds-to-minutes. Read replicas scale reads off primary but add async lag (RPO>0, eventual). For active/active DBs the anti-split-brain tool is quorum + fencing.

Introduction

What is this?

Managed HA: RDS/ Aurora Multi-AZ, Azure SQL DTU business-tier AG replicas (RTO seconds), GCP Cloud SQL HA. Read replicas: RDS (up to 15), Aurora replicas, Azure read scale-out, Cloud SQL read replicas; replication lag monitoring. Auto-failover sequence: health loss → promote standby → DNS flips (CNAME) → app reconnects. RPO: sync=0 (with edge cases), async=N seconds. Geodistributed DB patterns: Aurora Global, CosmosDB multi-region writes, Spanner (god-mode synchronous across regions). Self-managed clusters: PostgreSQL streaming + repmgr/Patroni (etcd quorum), MySQL group replication, MongoDB replica sets (majority vote; no write without majority). Split-brain: two writable primaries after partition → resolve with quorum + fencing (kill the loser + stepdown). Caching layers interplay (Redis replication) and warm pools for RTO.

Motivation

Why do you need it?

  • The DB is where outages actually hurt: it's the state you can't recreate from a dead node.
  • Managed multi-AZ buys real RPO=0 automatic failovers for the cost of a standby.
  • Misunderstanding quorum writes is how 'HA'd' databases split-brain and corrupt data.
  • Read replicas take the load off HA so a busy primary doesn't fail from capacity.
Think about it

Mental model

Imagine

Two welders at opposite ends of a bridge; if one vanishes the other must keep the structure — hence a third observer (supervisor) who votes on who is the lead

Think of

Quorum + primary election

The bridge (data) can only be built (written) under the supervisor's majority agreement. Without the third observer, the vanished welder and the surviving one each believe they lead — split-brain. Quorum prevents that.

Mechanics

How it works

  • Primary receives writes, streams WAL/binlog/commits to standby(s).
  • Standby applies them; sync mode requires the standby's confirmation before commit ACK.
  • Health monitor detects primary loss → majority (or managed runbook) promotes standby.
  • DNS/TLS endpoint flips; app reconnects with new primary.
  • In quorum DBs (Mongo/Oracle-like), writes also require a majority of voters to proceed.

Managed multi-AZ DB failover.

Primary (AZ-A)
Standby (AZ-B)
Loss of primary
Promote
DNS flip
Hands-on

Examples

Quorum pseudo-code (Patroni / etcd)

Quorum pseudo-code (Patroni / etcd)
# Patroni (PostgreSQL HA) uses an etcd/DCS quorum:
# - 2-node primary+replica + etcd learner = 3 voters
# - If primary loses contact, majority (2/3) elects replica
# - Writes from old primary are fenced: `patronictl overridden`/watchdog stops it.
# Useful ops: patronictl list — shows primary, replica, lag.

Majority of voters is the anti-split-brain: with 3 total, any partition must command 2 of them or writes die.

Real world

In production

  • RDS Multi-AZ for OLTP business services; Aurora for scale + read-replica elasticity.
  • Read replicas X2 behind a read-proxy; monitor replica lag < some seconds.
  • Mongo replica set with majority writeConcurrency + heartbeats for quorum.
Guidelines

Best practices

  • Use the managed failover, then test it — quarterly failover drills beat surprise ones.
  • Set the app's connection pool to refresh DNS + retry gracefully on failover.
  • Keep replica lag visible on dashboards; async drift silently eats you.
  • Quorum DBs: know your voter count and split-brain fence plan in writing.
  • Plan RTO acceptance: most teams accept 60-90s managed failovers, not 5s.
Watch out

Common mistakes

  • Two-node 'HA' without a quorum voter — a network blip creates two primaries (data corruption).
  • Read replicas counted as failover targets — they are async and may be stale (RPO>0).
  • Failover script that points SIGHUP/stop at the wrong process tree.
  • Skipping the failover drill that could've shown the DNS flip takes 90s on your provider.
When things break

Debugging tips

  • Failover happened; data seemed to appear 'lost' — was replication really sync? Check RPO cap and lag before failover.
  • Split-brain feared — run the majority-check first (healthy voters left?), then watchdog the old primary off.
  • App erroring immediately post-flip — the pool cached the old writer; set connection validation/DNS TTL.
  • Replica lag chronic — check WAL apply rate vs write load; add indexes on replica only if needed.
Interviews

Interview checkpoint

Practice

Hands-on lab

Failover a Localish Postgres Pair With Quorum
45 min · advanced
In progress
Objectives
  • Run a primary + replica via Patroni + etcd (3 voters)
  • Observe an auto-failover and fencing
  • Verify writes redirect with RPO=0 semantics in the lab

Failover a Localish Postgres Pair With Quorum

45 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A read replica can serve as your failover target because:

2

Split-brain in a two-node database is prevented by:

Go deeper

Official documentation

Amazon RDS Multi-AZ
docs
Patroni HA PostgreSQL
docs
MongoDB replica sets
docs