highAdvancedSymptom-first#gke#upgrade#cluster#rolling#maintenance

GKE cluster upgrade stuck or hung

gcloud container clusters list shows the cluster with state UPGRADING for a long time

Likely source:deploymentavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • gcloud container clusters list shows the cluster with state UPGRADING for a long time
  • New cluster version never becomes the running version
  • Node pools remain on the old version after a scheduled maintenance window
  • Changes to the cluster are rejected while it is busy upgrading

Plan an upgrade as two phases: master first, then node pools. If a rolling upgrade is stuck on a NotReady node, resolve or delete that node rather than repeatedly retrying the operation.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Cluster version and update state

    gcloud container clusters describe <CLUSTER> --zone <ZONE> --format='value(currentMasterVersion,status,statusMessage)'

  • Cluster operation log

    gcloud container operations list --filter='targetLink~<CLUSTER> AND operationType=UPDATE_MASTER' --format='table(name,status,progress,startTime,operationType)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Upgrade stalls at a specific node count
  • Auto-repair replaced nodes mid-upgrade
  • StatusMessage mentions node update stuck waiting on ready nodes

Confirm before fixing

gcloud container node-pools describe <POOL> --cluster <CLUSTER> --zone <ZONE> --format='value(management,status,statusMessage)'
READ

Read pool management and status during upgrade

Expected: Set of nodes mid-cordon or an explicit wait message

Fix

gcloud container node-pools update <POOL> --cluster <CLUSTER> --zone <ZONE> --enable-autoupgrade --enable-autorepair
CHANGE

Ensure auto-upgrade/repair is on so the update can proceed

Expected: Updated [<POOL>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud container operations list --filter='targetLink~<CLUSTER>' --format='table(name,operationType,status,error)'
READ

Read the stuck operation

2
kubectl get nodes -o wide
READ

Find NotReady nodes blocking the rolling update

3
gcloud container node-pools update <POOL> --cluster <CLUSTER> --zone <ZONE> --enable-autoupgrade --enable-autorepair
CHANGE

Ensure auto-upgrade/repair is on so the update can proceed

4
gcloud container clusters upgrade <CLUSTER> --zone <ZONE> --master
CHANGE

Force the control-plane version upgrade once blockers are cleared

Expected: Master of [<CLUSTER>] is being upgraded to <VERSION>.

Prevention

  • Enable maintenance windows + exclusions so releases do not collide with your upgrade.
  • Set maxSurge/maxUnavailable on pools to control rollout speed and headroom.
  • Keep PDBs tight enough to allow eviction but permissive enough to drain nodes.
  • Upgrade minor versions incrementally instead of jumping several at once.

Related runbooks

Same stack, same failure family — worth a skim.

GKE node pool unhealthy — nodes in UNKNOWN / NotReadycritical

gcloud container node-pools list shows node pool state UNKNOWN or unhealthy

Google CloudIntermediateUSE3 causes
Open runbook
GCE_INTERNAL_ERROR / zone capacity SERVICE_UNAVAILABLE on startcritical

gcloud compute instances start returns error GCE_INTERNAL_ERROR

Google CloudAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.