criticalFoundationUSE#disk#df#storage#filesystem

Disk full — filesystem at 100% usage

No space left on device errors in application logs

Likely source:availabilitydata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • No space left on device errors in application logs
  • Writes fail with ENOSPC
  • df -h shows 100% usage on one or more partitions

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Filesystem usage

    df -h

  • Largest directories

    du -sh /* 2>/dev/null | sort -rh | head -20

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • du shows /var/log dominating disk usage
  • A single log file is many gigabytes
  • logrotate is not configured or failing

Confirm before fixing

ls -lhS /var/log/*.log 2>/dev/null | head -10
READ

List log files sorted by size

Fix

truncate -s 0 /var/log/<logfile>.log
CHANGE

Truncate the oversized log file without restarting the service

Expected: File size becomes 0

systemctl restart rsyslog
CHANGE

Restart rsyslog so it reopens the file handle

Expected: rsyslog restarted

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
df -h
READ

Confirm which partition is full

2
du -sh /* 2>/dev/null | sort -rh | head -20
READ

Find the biggest directory consuming space

3
truncate -s 0 /var/log/<logfile>.log
CHANGE

Shrink the biggest log file (if logs are the cause)

4
apt-get clean 2>/dev/null || yum clean all 2>/dev/null
DESTRUCTIVE

Clear the package manager cache

Prevention

  • Configure logrotate for all application and system logs with size and rotation limits.
  • Alert on filesystem usage exceeding 80% via node_exporter metrics.
  • Schedule a weekly cron job to clean /var/log and package caches.
  • Use separate partitions for /var/log and /tmp to prevent a full log partition from bricking the rootfs.

Related runbooks

Same stack, same failure family — worth a skim.

Inodes exhausted — No space left on device despite free blockscritical

No space left on device but df -h shows free space

LinuxIntermediateUSE2 causes
Open runbook
Log files growing unbounded in /var/log consuming diskhigh

Disk usage climbing steadily over days/weeks

LinuxFoundationUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.