criticalProductionUSE#rds#storage#storage-full#outage

RDS in storage-full — database read-only or down

RDS status shows storage-full in the console

Likely source:availabilitydata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • RDS status shows storage-full in the console
  • Writes fail with 'disk full' / 'read-only' errors
  • DBInstanceStatus is storage-full, not available

When a multi-AZ RDS hits storage-full, the failover can also fail to start the secondary — grow storage BEFORE attempting failover.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Instance status

    aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].{Status:DBInstanceStatus,Allocated:AllocatedStorage,Class:DBInstanceClass}'

  • Storage metrics

    aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeStorageSpace --dimensions Name=DBInstanceIdentifier,Value=<DB-ID> --start-time <ISO> --end-time <ISO> --period 300 --statistics Minimum

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • FreeStorageSpace steady decay over days
  • No recent config change — organic data growth

Confirm before fixing

aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].{Allocated:AllocatedStorage,StorageType:StorageType,Max:MaxAllocatedStorage}'
READ

Current vs max allocation

Fix

aws rds modify-db-instance --db-instance-identifier <DB-ID> --allocated-storage <NEW-GB> --max-allocated-storage <NEW-MAX-GB> --apply-immediately
CHANGE

Grow the storage now (supports storage autoscaling on gp3/io1)

Expected: DBInstanceClass/Storage being modified

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].{Status:DBInstanceStatus,Allocated:AllocatedStorage}'
READ

Confirm storage-full

2
aws rds modify-db-instance --db-instance-identifier <DB-ID> --allocated-storage <NEW-GB> --max-allocated-storage <NEW-MAX-GB> --apply-immediately
CHANGE

Grow storage immediately

3
aws rds describe-db-instances --db-instance-identifier <DB-ID> --query 'DBInstances[0].DBInstanceStatus'
READ

Watch it return to available (then lease free space back with autoscaling)

Expected: available

Prevention

  • Enable storage autoscaling with a ceiling that tracks quota.
  • Alert on FreeStorageSpace under a two-day buffer (~20% of allocation).
  • Set binlog/WAL retention to match backup needs, not habit.

Related runbooks

Same stack, same failure family — worth a skim.

RDS max_connections reached — apps rejectedhigh

Apps error with 'too many connections' from the DB client

AWSIntermediateUSE3 causes
Open runbook
RDS read replica lagging behind the primaryhigh

Readers return stale data relative to the primary

AWSAdvancedRED3 causes
Open runbook
Multi-AZ RDS failover stuck in progresscritical

RDS status stuck in 'failover' or 'configuring-sync'

AWSAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.