High Availability
Intermediate 14 min read High Availability

DNS & Failover: Routing Around Trouble

Traffic follows health, not pride

DNS is the tip of failover: health-checked failover policies point traffic at healthy endpoints, while low TTLs keep the flip fast. Combine LB-level health checks with DNS-level secondary routing for layer-agnostic resilience.

DNS failoverhealth checkRoute 53TTLtraffic policyfailover recordendpoint
Prerequisites: DNS Routing Policies HA Fundamentals
Before we start

What you will learn

  • Design primary/secondary DNS failover with health checks
  • Weigh TTL vs. flip speed tradeoffs
  • Layer LB and DNS failover correctly
In 60 seconds

DNS failover records sit above the LB: primary points at the primary LB/IP, secondary at the fallback; a Route-53-style health check watches the primary and, when it fails, DNS answers with the secondary inside one TTL. DNS flips are global + slow-ish; LB flaps are intra-region + fast — so stack (fast local, slow remote) is the professional pattern.

Introduction

What is this?

Route 53: health checks + failover records (primary/secondary), weighted/routing policies; health check outputs to CloudWatch alarm hooks. Azure Traffic Manager (Priority / Weighted / Geographic / MultiValue profiles) + endpoint monitors; Google Cloud DNS + traffic policies (k8s ingress backends, internal LBs). TTL trade-off: low TTL (e.g., 30-60s) speeds flip + increases query load slightly; DNS whatever the resolver already cached at client/home routers — you cannot force a power-user's 86400-cache. Failover timing: health-check window (default ~30s) + TTL budget = flip in measured minutes, not seconds. LB first, DNS second; alternate-region failover when the LB/region itself is gone. also: ALB + Route53 (alias), health-check checklist covers the LB's IPs/endpoint, not just 'external reachable'. Connection draining interplay.

Motivation

Why do you need it?

  • When the region dies, only DNS can move traffic externally.
  • Customers with local caches are the worst case — TTL discipline is a CEO-era outage difference.
  • DNS failover duals the LB without double-billing a secondary data center.
  • DR-style cross-region shifts are DNS projects first.
Think about it

Mental model

Imagine

A building's main entrance that becomes a fire exit: signs (DNS) updated immediately because the door (LB) is out

Think of

DNS failover

Signs point where it's safe (healthy endpoint); the switch is as fast as the local updates (TTL). When the whole building is the fire (region), the signs matter even more — same principle, cross-region edition.

Mechanics

How it works

  • You define primary/secondary records plus a health-check per endpoint.
  • Route 53 evaluates the check; on failure the failover policy serves secondary.
  • Resolvers re-ask on their TTL window; clients pick up the flip progressively.
  • LB-level health checks meanwhile keep intra-region failover nearly instant.
  • When primary recovers, health check restores primary path (with warm-up logic carefully).

DNS failover records.

Users
Route 53
Primary
Secondary
Health check
Hands-on

Examples

Route53 failover record (pseudo)

Route53 failover record (pseudo)
# app.example.com (A record, alias)
#   Type: A alias → primary ALB (us-east-1)  / failover=PRIMARY   / health=chk-123
#   Type: A alias → secondary ALB (us-west-2) / failover=SECONDARY
# Health check chk-123: HTTP GET /healthz on ALB, interval 30s, threshold 3.
# TTL: 30s so the flip lands in ~a minute across resolvers.

Two aliases with one failover policy; traffic goes wherever the region is breathing.

Real world

In production

  • Belt-and-suspenders: ALB multi-AZ inside region + DNS failover to warm destination.
  • Traffic Manager Priority profile for Azure web apps in paired regions.
  • Health check on the real health endpoint so 'region alive but degraded' isn't answered by an IP ping.
Guidelines

Best practices

  • Point health checks at a synthetic 'am I really serving' endpoint (not 200-stub pages).
  • Keep TTLs stitched (30-60s) on records you intend to flip quickly.
  • Test the DNS flip end-to-end: change a health-check state and time till the answer changes.
  • Combine LB + DNS: LB for intra-region seconds, DNS for region-level minutes-long shifts.
  • Document warm secondaries: if you can't actually serve from secondary, the record is fiction.
Watch out

Common mistakes

  • Health check on an endpoint that never fails (static file) → the flip never happens.
  • TTL 3600 on your flip record + a regional outage = customers stuck for an hour.
  • Secondary points at an environment behind a VPN that cloud probes can't reach.
  • Assuming DNS flip is 5 seconds — budget 30s check window + TTL + client cache slop.
When things break

Debugging tips

  • Flip announced but customers still down — inspect edge/home resolver cache; document worst-case, drop TTL.
  • Health check 'degraded' noise — check the endpoint from the health-checker region (location-based per-view).
  • After recovery primary 'flappy' — the health window is too tight; lengthen threshold.
  • Traffic manager 'no endpoints healthy' — monitor profile health separately from record-level checks.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Health-Checked DNS Failover Pair
30 min · intermediate
In progress
Objectives
  • Configure primary/secondary target with a health check (Route 53 / Traffic Manager / dnsmasq sim)
  • Force a failure and observe the endpoint swap
  • Measure flip latency and tune TTL

Build a Health-Checked DNS Failover Pair

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

DNS-based failover is best for:

2

Your TTL strategy for flip-ready records:

Go deeper

Official documentation

Route 53 health checks & failover
docs
Azure Traffic Manager
docs
Cloud DNS docs
docs