highProductionUSE#vpc#subnet#ip-addresses#scaling

Subnet exhausted — no more IP addresses to launch

run-instances fails: There aren't enough IP addresses available in subnet

Likely source:scalingconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • run-instances fails: There aren't enough IP addresses available in subnet
  • ASG can't scale out even though CPU says it should
  • New ENIs fail to attach to instances

AWS reserves 5 addresses per subnet for infrastructure. AvailableIpAddressCount already excludes them, so treat 0 as truly 0. Fix IP exhaustion at the subnet/CIDR layer — not by shrinking instance sizes.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Available IP count

    aws ec2 describe-subnets --subnet-ids <SUBNET-ID> --query 'Subnets[0].{Cidr:CidrBlock,Available:AvailableIpAddressCount}'

  • Scaling failure output

    aws autoscaling describe-scaling-activities --auto-scaling-group-name <ASG-NAME> --max-items 5

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • AvailableIpAddressCount is 0 while instances maxed out the ASG
  • Subnet /28-/24 in a dense AZ

Confirm before fixing

aws ec2 describe-subnets --subnet-ids <SUBNET-ID> --query 'Subnets[0].{Cidr:CidrBlock,Available:AvailableIpAddressCount}'
READ

Confirm size vs availability

Fix

aws ec2 create-subnet --vpc-id <VPC-ID> --cidr-block <NEW-LARGER-CIDR> --availability-zone <AZ>
CHANGE

Create a larger subnet in the same AZ and add it to the ASG

Expected: subnet-...

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ec2 describe-subnets --subnet-ids <SUBNET-ID> --query 'Subnets[0].{Cidr:CidrBlock,Available:AvailableIpAddressCount}'
READ

Confirm exhaustion level

2
aws ec2 create-subnet --vpc-id <VPC-ID> --cidr-block <NEW-CIDR> --availability-zone <AZ>
CHANGE

Provision replacement capacity in the same AZ

3
aws autoscaling update-auto-scaling-group --auto-scaling-group-name <ASG-NAME> --vpc-zone-identifier "<EXISTING-SUBNET>,<NEW-SUBNET>"
CHANGE

Add the new subnet to the ASG

4
aws autoscaling set-desired-capacity --auto-scaling-group-name <ASG-NAME> --desired-capacity <N>
CHANGE

Trigger or allow scale-out to use the new capacity

Prevention

  • Size subnets for peak fleet + 20% (start /22 or larger for autoscaled tiers).
  • Alert when AvailableIpAddressCount drops below 10.
  • Separate lambda/LB/NAT into their own subnets so app scaling never starves them.

Related runbooks

Same stack, same failure family — worth a skim.

InsufficientInstanceCapacity — unable to launch or scalehigh

ec2 run-instances returns InsufficientInstanceCapacity

AWSProduction2 causes
Open runbook
Auto Scaling group not scaling out under loadhigh

Load climbs but instance count stays flat

AWSIntermediate4 causes
Open runbook
Private subnet without NAT — no outbound internethigh

Instances in private subnets cannot reach package repos or APIs

AWSFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.