highAdvancedUSE#load-average#d-state#iowait#uninterruptible#disk

Load average high with low CPU utilization — D-state processes

top shows load average well above CPU core count

Likely source:performanceavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • top shows load average well above CPU core count
  • CPU usage (%cpu) is low but load is 20+ on an 8-core system
  • top shows many processes in D (uninterruptible sleep) state

A high load average with low CPU is the classic D-state signature. The processes are not burning CPU — they are stuck in the kernel waiting for I/O that never completes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Load average and CPU state

    uptime && top -bn1 | head -5

  • D-state process count

    ps -eo state | grep -c '^D'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • D-state processes are waiting on block I/O (wchan shows blk_* or ext4_*)
  • iostat shows high await and low IOPS

Confirm before fixing

iostat -x 1 5
READ

Monitor disk I/O latency and throughput

Fix

iostat -x 1 3
READ

Identify the slow disk; replace or remount if hardware is failing

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
uptime && nproc
READ

Confirm load exceeds CPU cores

2
ps -eo state | grep -c '^D'
READ

Count D-state processes

3
ps -eo pid,stat,wchan:20,cmd | grep '^\s*[0-9].*D'
READ

Find what the D-state processes are waiting on

4
iostat -x 1 5
READ

Check disk I/O latency

Prevention

  • Monitor D-state process count via /proc/stat and alert when it exceeds 5.
  • Use fast local storage (SSD/NVMe) for I/O-heavy workloads.
  • Set NFS mount options: soft,timeo=30,retrans=3 to prevent indefinite hangs.
  • Alert on high iowait (>20%) via vmstat or node_exporter.

Related runbooks

Same stack, same failure family — worth a skim.

NFS mount hang — processes stuck in D-state on NFScritical

Processes stuck in D (uninterruptible sleep) state

LinuxAdvancedUSE3 causes
Open runbook
Runaway process pegging CPU — single process at 100%high

top shows one process at 100% CPU

LinuxFoundationUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.