criticalIntermediateUSE#inodes#df#filesystem#storage

Inodes exhausted — No space left on device despite free blocks

No space left on device but df -h shows free space

Likely source:availabilitydata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • No space left on device but df -h shows free space
  • df -i shows IUse% at 100%
  • mkdir or touch fails even though the disk is not full

Inode exhaustion is common on mail servers and session stores. The only long-term fix is preventing millions of tiny files from accumulating — not reformatting the disk.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Inode usage

    df -i

  • Directory with most files

    find / -xdev -type d -exec sh -c 'echo "$(find "$1" -maxdepth 1 | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head -10

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • find shows /var/spool/postfix, /tmp, or a session dir with millions of entries
  • df -h shows free blocks but df -i shows zero free inodes

Confirm before fixing

find /var/spool -xdev -type f | wc -l
READ

Count files in the spool directory

Fix

find /var/spool/postfix/maildrop -type f -mtime +7 -delete
DESTRUCTIVE

Remove old files from the queue directory

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
df -i
READ

Confirm inode exhaustion

2
find / -xdev -type d -exec sh -c 'echo "$(find "$1" -maxdepth 1 | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head -10
READ

Locate the directory with the most files

3
find <problem-dir> -type f -mtime +<N> -delete
DESTRUCTIVE

Delete old files in the offending directory

4
df -i
READ

Confirm inodes are freed

Prevention

  • Alert on inode usage exceeding 80% via node_exporter.
  • Use inotify or cron-based cleanup for directories that accumulate millions of small files.
  • Consider a separate /tmp partition with noexec and strict size limits.
  • Monitor application temp file creation rates.

Related runbooks

Same stack, same failure family — worth a skim.

Disk full — filesystem at 100% usagecritical

No space left on device errors in application logs

LinuxFoundationUSE3 causes
Open runbook
Log files growing unbounded in /var/log consuming diskhigh

Disk usage climbing steadily over days/weeks

LinuxFoundationUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.