Route53 health check false negatives
Health checks report unhealthy while the endpoint responds fine
Focused incident mode — strips away everything but this runbook.
Symptoms
- Health checks report unhealthy while the endpoint responds fine
- Traffic flaps between regions despite a stable service
- Health check status disagrees with your own monitoring
Evidence to grab
Collect these artifacts before you change anything — they prove (or rule out) each cause.
Health check settings
aws route53 get-health-check --health-check-id <HEALTH-CHECK-ID> --query 'HealthCheck.HealthCheckConfig'Live status
aws route53 get-health-check-status --health-check-id <HEALTH-CHECK-ID> --query 'HealthCheckObservations[0].StatusReport'
Diagnostic flow
Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.
Guided decision tree is ready.
Steps with commands, expected output, and pass/fail branches.
Ranked root causes
Evidence checklist
- All regions fail while direct curl works
- Endpoint has IP allowlists or Geo restrictions
Confirm before fixing
aws route53 get-health-check --health-check-id <HEALTH-CHECK-ID> --query 'HealthCheck.HealthCheckConfig.{Target:IPAddress|FullyQualifiedDomainName}'What does the check actually hit
Fix
aws route53 update-health-check --health-check-id <HEALTH-CHECK-ID> --ip-address <DIRECT-IP>Point the check at a directly routable IP (or add the Route53 checker ranges to the allowlist)
Resolution runbook
Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.
aws route53 get-health-check-status --health-check-id <HEALTH-CHECK-ID>See the last verdict and timestamps
curl -s -o /dev/null -w '%{http_code} %{time_total}' http://<TARGET><PATH>Independent reproduction
aws route53 update-health-check --health-check-id <HEALTH-CHECK-ID> --resource-path /healthz --ip-address <DIRECT-IP>Align the probe with a directly-checkable target
Prevention
- Register a stable /healthz on a public IP without geo/IP filtering.
- Set FailureThreshold to 3 and alarm-link health checks into Route53 records only.
- Cross-check once an hour with your own external checks (your internal p95 is not a truth source).
Related runbooks
Same stack, same failure family — worth a skim.
Primary region dies but DNS keeps answering with its address
One target shows unhealthy while others stay healthy
Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.