highAdvancedSymptom-first#aks#upgrade#kubernetes#provisioning#control-plane

AKS upgrade stuck — cluster not in Succeeded state

az aks show returns provisioningState = Upgrading for more than 30 minutes

Likely source:deploymentavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • az aks show returns provisioningState = Upgrading for more than 30 minutes
  • kubectl cluster-info shows the API server is responding but nodes are not upgrading
  • az aks get-upgrades shows the cluster is behind by multiple minor versions

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Cluster provisioning state

    az aks show -g <RG> -n <AKS> --query "{state:provisioningState,k8sVersion:kubernetesVersion}" -o json

  • Node pool upgrade status

    az aks nodepool list --resource-group <RG> --cluster-name <AKS> --query "[].{name:name,state:provisioningState}" -o table

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Cluster upgrade Succeeded but node pool upgrade stuck
  • PodDisruptionBudgets prevent cordoning nodes

Confirm before fixing

kubectl get pdb -A --no-headers
READ

List PDBs and their current disruption budget

Fix

az aks nodepool upgrade --resource-group <RG> --cluster-name <AKS> --name <POOL> --kubernetes-version <VERSION> --max-surge 33%
CHANGE

Upgrade the node pool with surge to bypass tight PDBs

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
az aks show -g <RG> -n <AKS> --query "{state:provisioningState,version:kubernetesVersion}" -o json
READ

Check cluster state

2
az aks nodepool list --resource-group <RG> --cluster-name <AKS> --query "[].{name:name,state:provisioningState}" -o table
READ

Per-pool status

3
az aks nodepool upgrade --resource-group <RG> --cluster-name <AKS> --name <POOL> --kubernetes-version <VERSION> --max-surge 33%
CHANGE

Manually trigger node pool upgrade with surge

4
kubectl get nodes -o wide
READ

Verify all nodes are on the new version

Prevention

  • Set upgradeChannel to stable and use auto-upgrade in non-production clusters.
  • Remove or loosen PDBs before a planned upgrade window.
  • Use max-surge to allow extra nodes during the upgrade roll.
  • Upgrade one minor version at a time and validate between each step.

Related runbooks

Same stack, same failure family — worth a skim.

AKS node pool NotReady — nodes failing health checkscritical

kubectl get nodes shows NotReady for nodes in a specific pool

AzureIntermediateUSE3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.