highIntermediateSymptom-first#vpc#nacl#security-group#firewall

NACL vs security group — connection dropped by the wrong layer

Traffic fails in one direction only (e.g. inbound works, replies dropped)

Likely source:connectivityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Traffic fails in one direction only (e.g. inbound works, replies dropped)
  • Changing the security group to 'allow all' does not fix connectivity
  • Test from a different IP range succeeds — grounding rules to specifics

The memory aid: NACL = stateless (allow both legs of every conversation), SG = stateful (allow the initiating direction). When 'SG allows everything' still fails, the NACL is guilty by default.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Subnet NACL entries

    aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<SUBNET-ID> --query 'NetworkAcls[0].Entries'

  • Instance security groups

    aws ec2 describe-instances --instance-ids <INSTANCE-ID> --query 'Reservations[0].Instances[0].SecurityGroups'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Inbound on 443 is allowed but clients time out after connect
  • NACL outbound lacks 1024-65535 for established responses

Confirm before fixing

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<SUBNET-ID> --query 'NetworkAcls[0].Entries[?Egress==`false`]'
READ

Check the egress direction

Fix

aws ec2 create-network-acl-entry --network-acl-id <ACL-ID> --egress --rule-number <N> --protocol tcp --rule-action allow --cidr-block 0.0.0.0/0 --port-range From=1024,To=65535
CHANGE

Add the ephemeral range to egress (and the reverse for inbound)

Expected: rule added

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<SUBNET-ID> --query 'NetworkAcls[0].Entries'
READ

Rule out the stateless layer first

2
aws ec2 describe-security-groups --group-ids <SECURITY-GROUP-ID> --query 'SecurityGroups[0].{Ingress:IpPermissions,Egress:IpPermissionsEgress}'
READ

Then the stateful layer

3
aws ec2 create-network-acl-entry --network-acl-id <ACL-ID> --egress --rule-number <N> --protocol tcp --rule-action allow --cidr-block 0.0.0.0/0 --port-range From=1024,To=65535
CHANGE

Fix the missing response-path rule

4
aws ec2 authorize-security-group-ingress --group-id <SECURITY-GROUP-ID> --protocol tcp --port <PORT> --cidr <CLIENT-CIDR>
CHANGE

Fix the missing SG rule when applicable

Prevention

  • Keep NACLs as emergency break-glass and make SGs the normal policy surface.
  • Include ephemeral-port egress in every NACL template.
  • Use VPC Reachability Analyzer in CI for any flow that crosses a NACL.

Related runbooks

Same stack, same failure family — worth a skim.

Public subnet instances have no internet accesscritical

ping/curl to the internet fails from an instance in a public subnet

AWSIntermediate3 causes
Open runbook
Private subnet without NAT — no outbound internethigh

Instances in private subnets cannot reach package repos or APIs

AWSFoundation3 causes
Open runbook
ALB target unhealthy — health check configuration drifthigh

One target shows unhealthy while others stay healthy

AWSIntermediateRED3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.