criticalAdvancedSymptom-first#rds#multiaz#failover#outage

Multi-AZ RDS failover stuck in progress

RDS status stuck in 'failover' or 'configuring-sync'

Likely source:availability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • RDS status stuck in 'failover' or 'configuring-sync'
  • Primary endpooint unreachable during an outage window
  • DNS endpoint does not resolve to the standby after reboot --force-failover

If a failover has not completed in ~15 minutes, you are past 'slow' and into 'blocked'. Storage-full and AZ capacity are the two causes that no amount of waiting fixes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Instance status and failover state

    aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].{Status:DBInstanceStatus,MultiAZ:MultiAZ,Modified:PendingModifiedValues}'

  • RDS events

    aws rds describe-events --source-type db-instance --source-identifier <DB-ID> --duration 360

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Storage-full condition detected before/during failover
  • Status reports storage issue in events

Confirm before fixing

aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].{Status:DBInstanceStatus,Allocated:AllocatedStorage}'
READ

Storage status check

Fix

aws rds modify-db-instance --db-instance-identifier <DB-ID> --allocated-storage <NEW-GB> --apply-immediately
CHANGE

Grow storage, which unblocks the standby bring-up

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].{Status:DBInstanceStatus,MultiAZ:MultiAZ}'
READ

Baseline the stall

2
aws rds describe-events --source-type db-instance --source-identifier <DB-ID> --duration 360 --query 'Events[].Message'
READ

Read the event trail

3
aws rds modify-db-instance --db-instance-identifier <DB-ID> --allocated-storage <NEW-GB> --apply-immediately
CHANGE

Grow storage if that is the blocker

4
nslookup <DB-ENDPOINT>
READ

Verify endpoint reconverges on the standby after promotion

Prevention

  • Keep storage autoscaling on so failover never collides with a full disk.
  • Test forced failover quarterly and time the RPO/RTO you actually get.
  • Run apps with a rediscovery loop so DNS endpoint moves are absorbed.

Related runbooks

Same stack, same failure family — worth a skim.

RDS in storage-full — database read-only or downcritical

RDS status shows storage-full in the console

AWSProductionUSE3 causes
Open runbook
RDS read replica lagging behind the primaryhigh

Readers return stale data relative to the primary

AWSAdvancedRED3 causes
Open runbook
Route53 failover not switching traffic to the standbycritical

Primary region dies but DNS keeps answering with its address

AWSProduction3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.