highAdvancedRED#gke#autoscaler#scaling#capacity#cluster

GKE cluster autoscaler not scaling nodes

Pending pods never get nodes even under sustained load

Likely source:scalingavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Pending pods never get nodes even under sustained load
  • Request rate climbs but node count stays flat
  • kubectl get pods shows many Pending with no scale-up resolution
  • gcloud container clusters describe shows no scale-up activity

Cluster autoscaler only reacts to Pending pods — it does not scale on CPU utilization alone. If pods are not pending, check horizontal pod autoscaling first; if they are pending but no nodes appear, the pool max or zone capacity is the blocker.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Scheduling events for the pending pods

    kubectl describe pod <POD> -n <NS> | grep -A15 Events

  • Autoscaler profile

    gcloud container clusters describe <CLUSTER> --zone <ZONE> --format='value(autoscaling)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Pool autoscaling max-node-count reached or unset
  • Autoscaler condition shows MaxNodesReached
  • Pending pods persist but the pool is full

Confirm before fixing

gcloud container node-pools describe <POOL> --cluster <CLUSTER> --zone <ZONE> --format='value(initialNodeCount,autoscaling)'
READ

Read pool node count and autoscaling limits

Expected: maxNodeCount equal to current count

Fix

gcloud container node-pools update <POOL> --cluster <CLUSTER> --zone <ZONE> --max-nodes=<NEW_MAX>
CHANGE

Raise the maximum node count

Expected: Updated [<POOL>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl describe pod <POD> -n <NS> | grep -A15 Events
READ

Confirm unschedulable reason

2
gcloud container clusters describe <CLUSTER> --zone <ZONE> --format='value(autoscaling.profile)'
READ

Verify autoscaling is enabled with limits

3
gcloud container node-pools update <POOL> --cluster <CLUSTER> --zone <ZONE> --max-nodes=<NEW_MAX>
CHANGE

Raise max nodes if at limit

4
kubectl get pods -A | grep Pending
READ

Confirm pending pods get scheduled after fix

Prevention

  • Set sane min/max on every pool and test scale-up during load tests.
  • Alert on pending pods and autoscaler TooManyErrors conditions.
  • Run pools in multiple zones so a single zone's capacity does not block scale-up.
  • Use budget-based HPA (requests) so pods become pending deterministically under load.

Related runbooks

Same stack, same failure family — worth a skim.

Pods stuck in Pending state — unschedulablehigh

kubectl get pods shows Pending for minutes

KubernetesIntermediate3 causes
Open runbook
GKE node pool unhealthy — nodes in UNKNOWN / NotReadycritical

gcloud container node-pools list shows node pool state UNKNOWN or unhealthy

Google CloudIntermediateUSE3 causes
Open runbook
GCE_INTERNAL_ERROR / zone capacity SERVICE_UNAVAILABLE on startcritical

gcloud compute instances start returns error GCE_INTERNAL_ERROR

Google CloudAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.