highIntermediateUSE#lsof#deleted#disk#file-descriptors

Deleted open file holding disk space — phantom disk usage

df shows full disk but du shows less total usage

Likely source:availabilitydata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

  • df shows full disk but du shows less total usage
  • lsof shows files with (deleted) suffix
  • Disk space does not reclaim after removing a large log file

Docker and systemd-journald are common culprits — both hold log file handles for the lifetime of the container/journald process. Use logrotate with copytruncate or restart containers after rotation.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Deleted files still open

    lsof +L1

  • Space delta between df and du

    df -h / && du -sh / 2>/dev/null

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • lsof shows a process holding a (deleted) file with GB-sized link count
  • The file was removed from /var/log but the process never restarted

Confirm before fixing

lsof +L1 2>/dev/null | grep deleted | sort -k7 -rn | head -5
READ

Find the largest deleted-but-open files

Fix

kill -HUP <PID>
CHANGE

Send HUP to the process so it closes the old file handle and reopens (if the app supports it)

Expected: The process reopens the log file; the deleted file space is freed

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
df -h / && du -sh / 2>/dev/null
READ

Confirm the space discrepancy

2
lsof +L1 2>/dev/null | sort -k7 -rn | head -10
READ

Find deleted files still held open

3
kill -HUP <PID>
CHANGE

Ask the process to reopen its log files

4
systemctl restart <service>
CHANGE

Restart if HUP does not release the handle

Prevention

  • Use logrotate with copytruncate for applications that cannot reopen file handles.
  • Alert on df vs du discrepancy via a cron job.
  • Restart services after log rotation if copytruncate is not an option.
  • Monitor lsof +L1 periodically in automated health checks.

Related runbooks

Same stack, same failure family — worth a skim.

Disk full — filesystem at 100% usagecritical

No space left on device errors in application logs

LinuxFoundationUSE3 causes
Open runbook
Binary deleted but process still running — live process from deleted filemedium

Upgraded binary but the running process still shows old version

LinuxIntermediate2 causes
Open runbook
Too many open files — EMFILE error on processhigh

EMFILE: Too many open files error in application logs

LinuxIntermediateUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.