highAdvancedUSE#nodes#eviction#pressure#capacity

Pods evicted — node pressure (disk, memory, pid)

Terminating/Evicted pod states on nodes

Likely source:availabilityscaling

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Terminating/Evicted pod states on nodes
  • Pods disappear abruptly, app goes down without a CrashLoop
  • describe shows reason=Evicted with message about node.kubernetes.io/disk-pressure

Repeated evictions of the SAME workload = workload too large for the node, not a cluster fault. Re-tune requests OR scale the node pool.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Evicted pod reason

    kubectl describe pod <pod> -n <namespace> | grep -A3 Status

  • Node pressure conditions

    kubectl describe node <node> | grep -A20 Conditions

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Node condition DiskPressure=True
  • df shows /var/lib/kubelet > 85% (default eviction threshold)

Confirm before fixing

ssh <node> df -h /var/lib/kubelet
READ

Confirm disk usage

Fix

ssh <node> docker system prune -af || ssh <node> crictl rmi --prune
DESTRUCTIVE

Remove dangling images/containers on the host

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl describe node <node> | grep -A20 Conditions
READ

Identify which pressure signal fired

2
ssh <node> df -h /var/lib/kubelet
READ

Disk path first

3
ssh <node> docker system prune -af --volumes || ssh <node> crictl rmi --prune
DESTRUCTIVE

Free space (verify before --volumes)

4
kubectl get pods -A -o wide | grep <node>
READ

Re-check affected workloads after cleanup

Prevention

  • Set requests/limits so pods sum below node allocatable (leaving headroom).
  • Alert on node conditions and kubelet eviction metrics.
  • Right-size nodes; use node pools with autoscaling for burst.
  • Adjust kubelet --eviction-hard for the workload and monitor image usage with periodic prune.

Related runbooks

Same stack, same failure family — worth a skim.

Worker node NotReady — node down or kubelet failurecritical

kubectl get nodes shows NotReady for one or more workers

KubernetesAdvancedUSE3 causes
Open runbook
OOMKilled containers — memory limit exceededcritical

Exit code 137 on the container

KubernetesFoundationUSE3 causes
Open runbook
High CPU throttling — containers starved by CPU limitsmedium

High response latency with low CPU saturation on the node

KubernetesAdvancedUSE2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.