criticalAdvancedUSE#nodes#kubelet#notready#capacity

Worker node NotReady — node down or kubelet failure

kubectl get nodes shows NotReady for one or more workers

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • kubectl get nodes shows NotReady for one or more workers
  • Pods on the node are rescheduled or stuck Terminating
  • DaemonSets (kube-proxy, node-exporter) report failures

Always get eyes on the HOST when a node is NotReady — the cluster will not tell you why the host is sick.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Node conditions

    kubectl describe node <node> | grep -A20 Conditions

  • Kubelet status

    systemctl status kubelet (host) — or ssh <node>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • systemctl shows kubelet failed/dead on the node
  • journalctl shows kubelet startup errors

Confirm before fixing

ssh <node> systemctl is-active kubelet
READ

Check service state

Fix

ssh <node> sudo systemctl restart kubelet
CHANGE

Restart kubelet

Expected: ------- kubelet restarted

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get nodes
READ

Confirm NotReady state and which nodes are affected

2
kubectl describe node <node> | grep -A20 Conditions
READ

Read condition reasons

3
ssh <node> systemctl status kubelet
READ

Check kubelet on the host

4
ssh <node> sudo systemctl restart kubelet
CHANGE

Restart kubelet if unhealthy

5
kubectl uncordon <node>
CHANGE

Allow scheduling again after recovery

Expected: node/<node> uncordoned

Prevention

  • Alert on node Ready=False and MemoryPressure/DiskPressure conditions.
  • Right-size and monitor node disks; keep /var/lib/kubelet short of 80%.
  • Use pod disruption budgets and node pools so a lost node never takes the app down.
  • Set taint based eviction (node.kubernetes.io/not-ready) and guarantee replicas with topology spread.

Related runbooks

Same stack, same failure family — worth a skim.

Pods evicted — node pressure (disk, memory, pid)high

Terminating/Evicted pod states on nodes

KubernetesAdvancedUSE3 causes
Open runbook
Pods stuck in Pending state — unschedulablehigh

kubectl get pods shows Pending for minutes

KubernetesIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.