criticalIntermediateUSE#gke#node pool#notready#unknown#cluster

GKE node pool unhealthy — nodes in UNKNOWN / NotReady

gcloud container node-pools list shows node pool state UNKNOWN or unhealthy

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • gcloud container node-pools list shows node pool state UNKNOWN or unhealthy
  • kubectl get nodes shows NotReady for multiple nodes
  • Pods on the affected nodes become Pending or are evicted
  • Cluster autoscaler cannot replace the unhealthy nodes

Separate a node that reports NotReady-False (reporting itself sick) from one reporting Unknown (stopped heartbeating). Unknown usually means the host is down or unreachable; False is often a local resource/kubelet issue.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Node pool status

    gcloud container node-pools describe <POOL> --cluster <CLUSTER> --zone <ZONE> --format='value(status,statusMessage)'

  • Relevant nodes and conditions

    gcloud container clusters get-credentials <CLUSTER> --zone <ZONE> && kubectl get nodes -o wide

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Pool status shows RUNNING but nodes are NotReady
  • kubelet heartbeats stop (Ready condition Unknown)
  • Auto-repair is disabled on the pool

Confirm before fixing

gcloud container node-pools describe <POOL> --cluster <CLUSTER> --zone <ZONE> --format='value(management.autoRepair)'
READ

Confirm auto-repair is enabled

Expected: false if auto-repair is off

Fix

gcloud container node-pools update <POOL> --cluster <CLUSTER> --zone <ZONE> --enable-autoupgrade --enable-autorepair
CHANGE

Enable auto-repair and auto-upgrade on the pool

Expected: Updated [<POOL>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud container node-pools describe <POOL> --cluster <CLUSTER> --zone <ZONE> --format='yaml(status,statusMessage,management)'
READ

Confirm the pool-level health and management flags

2
kubectl get nodes -l cloud.google.com/gke-nodepool=<POOL>
READ

Confirm which nodes are NotReady

3
gcloud container node-pools update <POOL> --cluster <CLUSTER> --zone <ZONE> --enable-autorepair --enable-autoupgrade
CHANGE

Enable auto-repair if it is off

4
kubectl get pods -A -o wide | grep -E 'Pending|Evicted'
READ

Confirm impacted workloads recover after repair

Prevention

  • Enable auto-repair and auto-upgrade on every node pool.
  • Run pools across multiple zones and keep node count >= 2 per zone.
  • Set PodDisruptionBudgets and inter-pod anti-affinity so pool repairs do not take the app down.
  • Alert on GKE node Ready=False and on pool status not RUNNING.

Related runbooks

Same stack, same failure family — worth a skim.

Worker node NotReady — node down or kubelet failurecritical

kubectl get nodes shows NotReady for one or more workers

KubernetesAdvancedUSE3 causes
Open runbook
GKE cluster autoscaler not scaling nodeshigh

Pending pods never get nodes even under sustained load

Google CloudAdvancedRED3 causes
Open runbook
Pods stuck in Pending state — unschedulablehigh

kubectl get pods shows Pending for minutes

KubernetesIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.