criticalIntermediateUSE#ec2#ebs#disk-full#capacity

Root EBS volume filling up — instance degraded

df -h / shows 95-100% usage on the root device

Likely source:availabilityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • df -h / shows 95-100% usage on the root device
  • Services crash with No space left on device
  • Core dumps, logs and temp files stop writing

Elastic Volumes let you grow online, but the partition/fs grow still happens on the host. Run the resize commands immediately after modify-volume completes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Filesystem usage

    aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["df -h /", "df -i /"]'

  • Volume size and type

    aws ec2 describe-volumes --volume-ids <VOLUME-ID> --query 'Volumes[0].{Size:Size,Type:VolumeType,State:State}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • du output shows /var/log or /tmp as the top grower
  • No log rotation OR rotation retained too little

Confirm before fixing

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["du -sh /var/log/* | sort -hr | head"]'
READ

Inspect the log directory spread

Fix

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["journalctl --vacuum-size=200M"]'
CHANGE

Vacuum journald before/while investigating the app

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ec2 describe-volumes --volume-ids <VOLUME-ID> --query 'Volumes[0].{Size:Size,Type:VolumeType,State:State}'
READ

Check the current volume configuration

2
aws ec2 modify-volume --volume-id <VOLUME-ID> --size <NEW-SIZE-GB> --region <REGION>
CHANGE

Grow the volume in place

3
aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters "commands=["ROOT=$(df --output=source / | tail -1)","growpart ${ROOT%[0-9]*} 1","resize2fs $ROOT"]"
CHANGE

Expand the partition and filesystem (ext4 example)

4
aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["df -h /"]'
READ

Verify the root filesystem now shows the new size

Expected: Total size matches the new volume size

Prevention

  • Alert when disk_used_bytes / filesystem > 80% (CloudWatch agent or Prometheus exporter).
  • Set logrotate/journald size caps tuned to disk size.
  • Size root volumes for headroom and test your resize runbook in staging.

Related runbooks

Same stack, same failure family — worth a skim.

Disk full — filesystem at 100% usagecritical

No space left on device errors in application logs

LinuxFoundationUSE3 causes
Open runbook
EBS snapshot stuck in 'creating' statemedium

aws ec2 describe-snapshots shows State: pending for hours

AWSAdvancedUSE3 causes
Open runbook
EBS volume stuck in attaching statehigh

aws ec2 describe-volumes shows State: attaching for minutes

AWSIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.