Multi-Region Architecture
Advanced 16 min read Multi-Region Architecture

Data Synchronization Across Regions

Copy the truth, carefully, everywhere

The data plumbing of multi-region: DB replication models (log-shipping, snapshot, synchronous), object replication (CRR), cache invalidation across regions, queue replay, and idempotent reconciliation. Every pattern trades latency, cost, and consistency; the measure is lag + conflicts + RPO.

replicationCRRlog shippingsnapshotstreamingcache invalidationreconciliationidempotency
Prerequisites: Active-Active Architecture
Before we start

What you will learn

  • Map DB replication models to consistency/cost
  • Design cross-region object + cache sync
  • Implement idempotent replay for queues/events
In 60 seconds

DB sync: logical/log-shipping (RDS replica, Postgres streaming, Aurora), spatial multi-master (Cosmos, Spanner). Objects: S3/GCS cross-region replication (async). Cache: keep per-region, stateless; invalidate via versioning. Queues/events: replay/idempotent consumers. Reconciliation: periodic cross-region diff + repair.

Introduction

What is this?

DB: log-based (WAL/binlog shipping: lag seconds), Aurora cluster (storage-level replication sub-second), Spanner/Cosmos multi-region (strong/async multi-master). Object: AWS S3 CRR (async, regions pairs), GCS two regional pools OIDC, Azure GRS. File/protocol: S3 Transfer plus. Cache: per-region caches (memcache/redis cluster), versioned-key invalidation, caching-edge vs DB replication interplay. Queues: SQS cross-region not native — use DLQ replay, publisher fan-out via EventBridge to multiple regions, or a replicated Kafka (MirrorMaker) — and idempotent consumers (event+idempotency key in DB) so replays don't double-apply. Reconciliation: periodic checksum-joins between regions (batch diff + repair), conflict logs. Metrics: lag (replica apply gap), conflicts/hr, RPO achieved, divergence scan results, egress cost/MB.

Motivation

Why do you need it?

  • The RPO promise lives in replication lag; tracking it is the honesty of every DR/phrase.
  • CRR/object sync means user uploads survive region loss — the sneaky outage data people forget.
  • Idempotent replay is what makes retries safe — both in queues and post-DR cutover.
  • Reconciliation is the safety net for strict-metric systems; a drill without a diff can miss bleeding.
Think about it

Mental model

Imagine

Two ledgers, one in each office, both recording the same sales — one copies by messenger each night (async), the other by wire instantly (sync)

Think of

Replication models

The daily messenger (async log shipping) means the second office lags a day; the instant wire (sync/global DB) means both agree within milliseconds but the wire itself must be bulletproof. Pick the messenger or the wire by what the numbers demand.

Mechanics

How it works

  • DB log streams from primary to replica(s); lag meters the pipeline.
  • Objects/buckets replicate async via service, with versioning for safety.
  • Services use idempotent writes + replay (DLQ/EventBridge re-drive) for queue events.
  • Caches are per-region and stateless; versioned keys invalidate across regions.
  • Reconciliation job diffs regions and repairs divergence; metrics published.

Cross-region data sync.

Primary writes
Log shipping
Object CRR
Events
Reconcile
Hands-on

Examples

Idempotent consumer key

Idempotent consumer key
txn = { idempotency_key: "order-123-retry-7", amount: 42 }
# Consumer:
#   if not exists(processed, idempotency_key):
#       apply(txn)
#       mark(processed, key, now())
#   else: skip (already applied)
# → replay-safe across regions AND after DR cutover.

Idempotency keys turn 'replay this queue/event' from dangerous into default-safe — the unlock for cross-region retries.

Real world

In production

  • RDS cross-region replica + lag dashboard for the RPO reading.
  • S3 CRR on user uploads + versioning; CDN cache keys with versioned assets.
  • Reconciliation job nightly: checksum-join core tables across regions, repair conflicts.
Guidelines

Best practices

  • Instrument lag, conflict counts, and divergence every schedule — they're the RPO health panel.
  • Design for replay: event IDs + idempotent consumers to make cross-region retries boring.
  • Keep cache stateless + version-keyed; never treat cross-region cache as authoritative.
  • Reconcile periodically even in single-writer models (drift happens unannounced).
  • Watch egress cost per region: replication bandwidth is a real line on the bill.
Watch out

Common mistakes

  • A sync-replication DB across the ocean: every write pays the far-round-trip — latency tax the log model avoids.
  • Objects only in the primary region; uploads vanish when it dies.
  • Retries breaking idempotency → double-apply on replay (the whole point of keys).
  • Reconciliation absent until a drill: divergence caught at cutover time, not daily.
When things break

Debugging tips

  • Replica lag spikes → check network path/replication config, or a large DDL; tune channel + monitor.
  • Conflict counts climbing → your ownership rule vs LWW disagreement surging; review schema division.
  • Reconciliation repairs but floods logs → the diff job's predicate is wrong (e.g., comparing upd fields that legitimately differ).
  • Cross-region replay applies twice → idempotency key missing/batched weirdly; add the key + dedupe.
Interviews

Interview checkpoint

Practice

Hands-on lab

Sync Two 'Regions' and Reconcile
35 min · advanced
In progress
Objectives
  • Stream writes from primary to a replica (log-based)
  • Simulate a conflict + reconcile it
  • Watch lag as the RPO proxy

Sync Two 'Regions' and Reconcile

35 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Synchronous cross-region DB replication costs you:

2

The idempotency-key pattern's payoff:

Go deeper

Official documentation

S3 cross-region replication
docs
GCS replication
docs
EventBridge multi-region patterns
docs