DNS
Advanced 12 min read DNS

DNS Health Checks and Failover

Making DNS aware of the living world

DNS is stateless and can't know a server is down — so managed DNS adds health checks that probe endpoints and exclude unhealthy records from answers, powering region-level failover.

health checkfailoverprobeprimarysecondaryflags
Prerequisites: TTL and Caching
Before we start

What you will learn

  • Create health checks against endpoints
  • Wire health checks into failover records
  • Tune detection thresholds
In 60 seconds

A health check probes a URL/IP from multiple global vantage points; after a failure threshold, the record is removed from DNS answers and failover records shift traffic to the secondary — DNS turns down-outages into redirects.

Introduction

What is this?

Route 53 health checks: HTTP/HTTPS/TCP probes, configurable path, port, interval (10-30s), failure threshold, string matching, and 'calculated health checks' (combine child statuses). Azure Traffic Manager monitors endpoints likewise; GCP external health checks gate backends. Endpoint types: LB DNS, bucket, instance IP. Failover records: PRIMARY/SECONDARY pairs serve the healthy one via health check state.

Motivation

Why do you need it?

  • Clients never learn an endpoint is down at the DNS layer unless we tell them.
  • Health-based DNS is the first line of multi-region failover.
  • Alerts + health == a closed loop: if a region fails, both traffic and on-call get notified.
  • Shared probe vantage points catch 'works for you but not for users' issues.
Think about it

Mental model

Imagine

A cinema's 'now showing / today closed' sign triggered by a daily staff check

Think of

DNS health checks

The sign (records) never guesses — a person (probe) visits to verify the doors (endpoint) weekly-minute, SIGNS the sign on failure, and moviegoers reroute to the next cinema (failover).

Mechanics

How it works

  • Probes hit the endpoint from several vantage points every 30s by default.
  • Health = pass/fail per probe; threshold of N consecutive failures flips it unhealthy.
  • A record attached to an unhealthy check is not served to resolvers.
  • Failover pairs: PRIMARY record served while healthy; on failure the SECONDARY answers.
  • Timers: interval × threshold + TTL bounds the end-to-end failover time.

Failover timeline.

Primary healthy
Region down
Unhealthy
Failover
Recovery
Hands-on

Examples

Failover pair with health check

Failover pair with health check
# health check for primary
aws route53 create-health-check --caller-reference primary-hc \
  --health-check-config '{"Type":"HTTPS","ResourcePath":"/healthz","Port":443,"FailureThreshold":3,"RequestInterval":10}'
# PRIMARY + SECONDARY records
aws route53 change-resource-record-sets --hosted-zone-id ZONE \
  --change-batch '{"Changes":[
    {"Action":"UPSERT","ResourceRecordSet":{"Name":"app.example.com","Type":"A",
      "SetIdentifier":"primary","Failover":"PRIMARY","HealthCheckId":"hc-primary","TTL":60,
      "AliasTarget":{"HostedZoneId":"...","DNSName":"primary-lb"}}},
    {"Action":"UPSERT","ResourceRecordSet":{"Name":"app.example.com","Type":"A",
      "SetIdentifier":"dr","Failover":"SECONDARY","TTL":60,
      "AliasTarget":{"HostedZoneId":"...","DNSName":"dr-lb"}}}]}'

Probe the primary's /healthz; on failure DNS serves the secondary — the region-level circuit breaker.

Real world

In production

  • Two-region e-commerce: PRIMARY health check on us-east-1; SECONDARY record on eu-west-1 with TTL 60.
  • Azure global load balancer priority profile gates each region endpoint with probe cadence.
  • Calculated health checks require 2 of 3 region endpoints before failing over (majority-decision).
Guidelines

Best practices

  • Probe a path that reflects actual availability, not just TCP accept.
  • Set FailureThreshold so a single probe blip doesn't failover the world.
  • Keep the secondary TTL low too; failback must be deliberate, not flapping.
  • Add alerting to the health check transitions to know when DNS actually rerouted.
Watch out

Common mistakes

  • Health check on '/' when the real dependency is /healthz — missing the outage.
  • Threshold 1 → flapping failovers on transient 503s.
  • No alert on health-state change: DNS rerouted silently.
  • Failback (PRIMARY re-passes) without a steady-state; traffic flaps between regions.
Root cause analysis

Common issues

Error
Region failed but DNS still serves it
Root cause

Health check unhealthy but TTL/cache served the old record

Fix

Lower TTL on failover records and confirm the check flipped before expecting failover.

Error
Failover flapped 4 times/hour
Root cause

Threshold too low or probe path flaky

Fix

Increase FailureThreshold + interval; fix the probe dependency.

When things break

Debugging tips

  • 'Failover didn't happen': check the health check state directly, then TTL.
  • A region that probed fine but apps were down = the probe didn't exercise the real path — widen it.
  • Flapping between regions: raise thresholds and add a 'recovery' delay.
  • Verify from a probe region perspective: probe source locates the failure geographically.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build and Qualify a DNS Failover Design
25 min · advanced
In progress
Objectives
  • Wire a health-check + failover pair conceptually
  • Simulate the failover timer math
  • Practice the postmortem flow

Build and Qualify a DNS Failover Design

25 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/1
1

Which sequencing bounds the end-to-end DNS failover time?

Go deeper

Official documentation

AWS — Route 53 health checks
docs
Azure — Traffic Manager monitoring
docs
GCP — Health checks for LB
docs