criticalAdvancedUSE#aks#pod-cidr#ip-exhaustion#networking#scaling

AKS pod IP exhaustion — pod CIDR depleted

New pods stuck in Pending with FailedScheduling: no IP addresses available

Likely source:scalingconnectivity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • New pods stuck in Pending with FailedScheduling: no IP addresses available
  • kubectl get nodes reports pods unable to be scheduled on any node
  • Cluster auto-scaler cannot add nodes because the CIDR is full

Changing the pod CIDR on an existing AKS cluster is not possible without re-creation. Plan CIDR sizing upfront with room for growth.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Pod CIDR per node

    kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

  • AKS network profile

    az aks show -g <RG> -n <AKS> --query "{podCidr:networkProfile.podCidr,maxPods:agentPoolProfiles[0].maxPods}" -o json

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • podCidr = 10.244.0.0/16 only allows 256 nodes
  • Cluster scaled up beyond the CIDR capacity

Confirm before fixing

kubectl get nodes --no-headers | wc -l
READ

Count nodes

Fix

az aks nodepool scale --resource-group <RG> --cluster-name <AKS> --name <POOL> --node-count <REDUCED-COUNT>
CHANGE

Scale down the node pool to fit within CIDR

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
kubectl get pods -A -o wide --field-selector=status.phase=Running | awk '{print $8}' | sort | uniq -c | sort -rn
READ

Pods per node distribution

2
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
READ

Show pod CIDRs in use

3
az aks show -g <RG> -n <AKS> --query "{podCidr:networkProfile.podCidr,maxPods:agentPoolProfiles[0].maxPods}" -o json
READ

Review the network profile

4
az aks nodepool scale --resource-group <RG> --cluster-name <AKS> --name <POOL> --node-count <COUNT>
CHANGE

Scale down to reduce pod density

Prevention

  • Use a /14 or larger pod CIDR for clusters expected to scale beyond 20 nodes.
  • Set maxPods to 110 (default for Linux) on all node pools.
  • Monitor pod CIDR utilization with Azure Monitor and alert at 80 percent.
  • Use Azure CNI with overlay mode for better IP efficiency.

Related runbooks

Same stack, same failure family — worth a skim.

AKS node pool NotReady — nodes failing health checkscritical

kubectl get nodes shows NotReady for nodes in a specific pool

AzureIntermediateUSE3 causes
Open runbook
Pods stuck in Pending state — unschedulablehigh

kubectl get pods shows Pending for minutes

KubernetesIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.