highFoundationUSE#logs#var-log#logrotate#disk#rotation

Log files growing unbounded in /var/log consuming disk

Disk usage climbing steadily over days/weeks

Likely source:availabilitydata-loss

Focused incident mode — strips away everything but this runbook.

Symptoms

Disk usage climbing steadily over days/weeks
du shows /var/log growing with large .log files
No log rotation happening despite logrotate being installed

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

Largest log files
du -sh /var/log/* 2>/dev/null | sort -rh | head -10
Logrotate configuration
ls -la /etc/logrotate.d/ && cat /var/lib/logrotate/status

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

No entry in /etc/logrotate.d/ for the service
Log file has grown to many gigabytes without rotation

Confirm before fixing

ls /etc/logrotate.d/ | grep -i <service>

READ

Check for a logrotate config file

Fix

cat > /etc/logrotate.d/<service> << 'EOF'
/var/log/<service>/*.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
  create 0640 <user> <group>
}
EOF

CHANGE

Create a logrotate config for the service

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

du -sh /var/log/* 2>/dev/null | sort -rh | head -10

READ

Identify the largest log files

ls /etc/logrotate.d/ | grep -i <service>

READ

Check if logrotate is configured

truncate -s 0 /var/log/<logfile>.log

CHANGE

Shrink the oversized log file immediately

cat > /etc/logrotate.d/<service> << 'EOF'
/var/log/<service>/*.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
}
EOF

CHANGE

Create a logrotate config for the service

Prevention

Configure logrotate for every service that writes to /var/log during provisioning.
Alert on /var/log directory size exceeding 5GB.
Use journald with SystemMaxUse= to cap systemd journal size.
Set application log levels to INFO or WARN in production.

Related runbooks

Same stack, same failure family — worth a skim.

Disk full — filesystem at 100% usagecritical

No space left on device errors in application logs

LinuxFoundationUSE3 causes

Open runbook

Inodes exhausted — No space left on device despite free blockscritical

No space left on device but df -h shows free space

LinuxIntermediateUSE2 causes

Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.