Active-Active Multi-Region
Every user's region is the right one — the data is the hard part
Two (or N) regions serving simultaneously: users hit their local region, writes replicate everywhere, and you must resolve concurrent writes. Earned only with near-zero RTO needs + huge user spread. Conflict strategy and sharding decide whether it's brilliant or a nightmare.
What you will learn
- Determine when active-active is worth it
- Manage concurrent multi-master writes (conflict rules, sharding)
- Design reads-local + reconcile writes coherently
Active-active = same logical app serving from ≥2 regions, each able to accept writes. Data must reconcile across regions: single-master plus replicas (AGAINST true AA), multi-master DBs (CosmosDB multi-region writes, Aurora Global secondary-rw, Spanner), or shard-by-user (strategy: region owns its shards, occasional cross-region). Conflict rule (LWW, provenance, shard ownership) is decided upfront.
What is this?
Data models: 1) single-master (primary region writes, others replicate read) — NOT AA, but 'hot active' reads; 2) multi-master with conflict resolution — CosmosDB multi-write regions (LWW + custom conflict resolvers), DynastorDB Global Tables (last-writer-wins; async), Spanner (external synchronous replication, strong consistency — true AA with strong reads), CockroachDB (reads AQ(), writes external); 3) partitioned/shared data — user → home region (shard ownership), cross-region occasional reads/merges. Consistency reality: multi-master writes → eventual consistency windows + conflict remedy; capacity/coherence engineering needed. App layer: region-local reads (replica-first), idempotent writes with conflict markers, region-aware connection pools, per-region feature assets. RPO/RTO: AA's payoff — region loss ≈ traffic shift, near-zero downtime; the price is high (data replication cost, bandwidth egress, ops duplication, conflicting-writes discipline). Gateway/Axact/Spanner hyper-specials; honest answer: most orgs don't truly need AA, they need reads-local + a promoted standby.
Why do you need it?
- For true global 24/7 apps (gaming, collab, high-stakes) the 'never down by region' payoff is real money.
- Reads-local + writes-when-possible to home region is the 'AA-lite' most apps actually achieve.
- Conflict strategy forces clarity: you face LWW + provenance or shard ownership — decide now, not at peak.
- AA at 2 regions teaches the disciplines (reconciliation, ops duplication) that make global architecture speak.
Mental model
Two bank branches in two cities, same customer: deposits at either must land in one ledger
Multi-master writes
Both branches happily take deposits — impossible without rules for 'which deposit wins' when both touch the same account at once. A sharding rule (account → home branch) gets around it; otherwise LWW + reconciliation decide.
How it works
- Deploy identical stacks in ≥2 regions behind global LB/steering.
- Route users to the nearest; both accept writes for their share.
- Data replicates across regions (async or Spanner-sync); reads local, writes eventually consistent (unless strong-consistency DB).
- Conflict policy runs on replicate autonomously (LWW / provenance / shard ownership).
- Region loss: traffic shifts to siblings; near-zero downtime (possibly stale-but-reconciled data briefly).
Active-active topology.
Examples
Conflict resolution stub (LWW provenance)
// e.g., CosmosDB multi-master LWW + custom resolver// Rule: winning write = highest (version, region.tiebreak) seen.// For sharded ownership: userId hash % N → that region is authoritative for the row;// other regions treat writes as reads-then-forward. // Practical: same row edited in two regions -> the later/rightful owner wins, log merges.Pick one rule you can DEFEND operationally: LWW with provenance, or shard ownership where regions truly divide the table.
In production
- CosmosDB multi-region writes with custom conflict resolver for user profiles.
- Spanner multi-region for truly strong-consistency global reads/writes.
- Sharded ownership: user home-region owns writes; cross-region reads legal-but-read-only.
Best practices
- Decide + document the conflict rule BEFORE multi-master; test it under concurrent writes.
- Start 'AA-lite': reads-local + home-region writes + promoted standby; upgrade only when latency/law demand.
- Idempotent writes + provenance stamps at the app edge.
- Monitor replication lag per region and conflict counts — those are your AA health metrics.
- Ops parity: same deploys, dashboards, and drills for both regions.
Common mistakes
- AA with single-master DB: you're paying double infra for reads only (that's 'hot standby', not AA).
- Two masters writing the same logical rows with no rule — corruption by LWW surprises.
- Ignoring synchronous replica latency: users get far-region write waits, or worse strong-consistency blocks.
- Ops drift: region B falls behind on releases/features → disasters only in one half.
Debugging tips
- Same row diverges between regions → check conflict resolution actually fired; add conflict-count alerting.
- Write latency spiking in one region → replication pipe saturated or sync-consistency contention; inspect cross-region egress.
- A region's reads show stale data → replica lag + steering sending reads that should go home-region.
- Failover worked but data mismatched → the reconciliation didn't catch the window; review ordering of ownership handoff.
Interview checkpoint
Quick quiz
The AA trap most teams hit first:
AA's health metrics you must dashboard: