highFoundationSymptom-first#vpc#nat#private-subnet#routing

Private subnet without NAT — no outbound internet

Instances in private subnets cannot reach package repos or APIs

Likely source:connectivityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Instances in private subnets cannot reach package repos or APIs
  • SSM/AWS APIs unreachable from private instances without VPC endpoints
  • curl to https://aws.amazon.com hangs while VPC-internal traffic works

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Routes for the private subnet

    aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<SUBNET-ID> --query 'RouteTables[0].Routes'

  • NAT gateway inventory

    aws ec2 describe-nat-gateways --filter Name=state,Values=available --query 'NatGateways[].{Id:NatGatewayId,Subnet:SubnetId,State:State}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • describe-nat-gateways returns empty (or only failed/deleting)
  • No default route anywhere

Confirm before fixing

aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=<VPC-ID> --query 'NatGateways[].{Id:NatGatewayId,State:State,Subnet:SubnetId}'
READ

Are there any NAT gateways in this VPC?

Fix

aws ec2 allocate-address --domain vpc && aws ec2 create-nat-gateway --subnet-id <PUBLIC-SUBNET-ID> --allocation-id <ALLOCATION-ID>
CHANGE

Create a NAT GW in a public subnet (requires an EIP)

Expected: nat-0...

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=<VPC-ID> --query 'NatGateways[].{Id:NatGatewayId,State:State}'
READ

Confirm the NAT inventory

2
aws ec2 allocate-address --domain vpc
CHANGE

Allocate an EIP for the NAT if none exists

3
aws ec2 create-nat-gateway --subnet-id <PUBLIC-SUBNET-ID> --allocation-id <ALLOCATION-ID>
CHANGE

Create the NAT gateway (wait for available)

Expected: nat-...

4
aws ec2 create-route --route-table-id <ROUTE-TABLE-ID> --destination-cidr-block 0.0.0.0/0 --nat-gateway-id <NAT-GATEWAY-ID>
CHANGE

Route private traffic through it

5
aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["curl -sI https://aws.amazon.com/ | head -1"]'
READ

Verify egress restored

Expected: HTTP/2 200 (or similar)

Prevention

  • Provision NAT gateways per-AZ in infrastructure code with route table exports.
  • Consider VPC endpoints for AWS APIs to avoid NAT altogether.
  • Add a drift check: every private route table must expose a default route.

Related runbooks

Same stack, same failure family — worth a skim.

Public subnet instances have no internet accesscritical

ping/curl to the internet fails from an instance in a public subnet

AWSIntermediate3 causes
Open runbook
Subnet exhausted — no more IP addresses to launchhigh

run-instances fails: There aren't enough IP addresses available in subnet

AWSProductionUSE3 causes
Open runbook
NACL vs security group — connection dropped by the wrong layerhigh

Traffic fails in one direction only (e.g. inbound works, replies dropped)

AWSIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.