criticalProductionSymptom-first#route53#failover#dns#health-check

Route53 failover not switching traffic to the standby

Primary region dies but DNS keeps answering with its address

Likely source:availabilityconnectivity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Primary region dies but DNS keeps answering with its address
  • Secondary region healthy but never receives traffic
  • Failover policy records exist but the cutover never happens

Route53 failover is only as honest as its health check. A primary whose check tracks only TCP:443 flips green while the app 500s — make the check reflect the dependency the outage actually breaks.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Records and their health-check linkage

    aws route53 list-resource-record-sets --hosted-zone-id <ZONE-ID> --query 'ResourceRecordSets[?Failover!=`null`].{Name:Name,Type:Type,Failover:Failover,HealthCheckId:HealthCheckId}'

  • Health check status

    aws route53 get-health-check-status --health-check-id <HEALTH-CHECK-ID> --query 'HealthCheckObservations[0].StatusReport'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • get-health-check-status returns Success while app is down
  • Check probes a path that doesn't exercise the dependency

Confirm before fixing

aws route53 get-health-check --health-check-id <HEALTH-CHECK-ID> --query 'HealthCheck.HealthCheckConfig'
READ

What the check actually verifies

Fix

aws route53 update-health-check --health-check-id <HEALTH-CHECK-ID> --resource-path /healthz --failure-threshold 3
CHANGE

Tighten the probe so 'app down' = 'check down'

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws route53 list-resource-record-sets --hosted-zone-id <ZONE-ID> --query 'ResourceRecordSets[?Failover!=`null`].{Name:Name,Type:Type,Failover:Failover,HealthCheckId:HealthCheckId}'
READ

Map the failover pair and associations

2
aws route53 get-health-check-status --health-check-id <HEALTH-CHECK-ID> --query 'HealthCheckObservations[0].StatusReport'
READ

Is the primary REALLY unhealthy?

3
dig +short <NAME> <TYPE> @8.8.8.8
READ

Current DNS answer

4
aws route53 change-resource-record-sets --hosted-zone-id <ZONE-ID> --change-batch file://fixed-batch.json
CHANGE

Wire the health checks to the records and correct Failover values

Prevention

  • Test failover quarterly with a forced primary outage, not a simulated one.
  • Point failover checks at deep paths that exercise the real stack.
  • Keep a runbook rumble: record TTLs appropriately (low) so failover actually moves clients fast.

Related runbooks

Same stack, same failure family — worth a skim.

Route53 health check false negativeshigh

Health checks report unhealthy while the endpoint responds fine

AWSAdvanced3 causes
Open runbook
Multi-AZ RDS failover stuck in progresscritical

RDS status stuck in 'failover' or 'configuring-sync'

AWSAdvanced3 causes
Open runbook
ALB returns 503 — no healthy targetscritical

HTTP 503 Service Unavailable from the load balancer

AWSIntermediateRED3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.