highAdvancedUSE#gcp#firewall#cloud nat#snat#networking

GCP firewall / Cloud NAT — SNAT exhaustion or traffic blocked

Outbound requests from VMs time out while inbound to the VPC works

Likely source:connectivityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Outbound requests from VMs time out while inbound to the VPC works
  • Cloud NAT logs show SNAT connection limit exceeded
  • App calls to external APIs fail intermittently with connection timeout
  • New firewall rule allows traffic but packets still dropped

Work egress and ingress separately: NAT for out, firewall for in. SNAT exhaustion shows up as flaky outbound timeouts and is fixed with more NAT IPs, not more firewall rules.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • NAT gateway health and IP count

    gcloud compute routers describe <ROUTER> --region <REGION> --format='value(nats)'

  • Firewall rules on the VPC

    gcloud compute firewall-rules list --filter='network=<NETWORK>' --format='table(name,direction,disabled,allowed,denied)'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • SSH/curl from VM to external endpoints intermittently times out
  • NAT metrics security_ports_allocated saturated
  • Only a NAT IP or two configured for a busy subnet

Confirm before fixing

gcloud compute routers describe <ROUTER> --region <REGION> --format='value(nats[0].natIps,nats[0].sourceSubnetworkIpRangesToNat)'
READ

Count NAT IPs and subnets being NATed

Expected: Fewer NAT IPs than the concurrent connection profile needs

Fix

gcloud compute routers update <ROUTER> --region <REGION> --nat-ip-allocation=dynamic --enable-nat-port-sampling
CHANGE

Use dynamic NAT IP allocation and enable port sampling for visibility

Expected: Updated [<ROUTER>].

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
gcloud compute ssh <INSTANCE> --zone=<ZONE> --command='curl -v --max-time 10 https://storage.googleapis.com/'
READ

Isolate the failing direction first

2
gcloud compute routers describe <ROUTER> --region <REGION> --format='json(nats)'
READ

Check NAT IP count and port allocation

3
gcloud compute routers update <ROUTER> --region <REGION> --nat-ip-allocation=dynamic
CHANGE

Scale NAT IPs if SNAT is exhausted

4
gcloud compute firewall-rules create <RULE> --direction=INGRESS --network=<NETWORK> --priority=<PRIORITY> --action=ALLOW --rules=tcp:443 --source-ranges=<CIDR>
CHANGE

Add the missing allow rule

5
curl -v --max-time 10 https://storage.googleapis.com/
READ

Retest egress after the fix

Prevention

  • Allocate enough Cloud NAT IPs for peak concurrent connections (1 IP ≈ 64k ports).
  • Monitor NAT ports-allocated ratio and alert before exhaustion.
  • Put firewall rule priorities and tags in code to avoid drift.
  • Right-size VPC subnets and secondary ranges long before GKE/VM growth hits them.

Related runbooks

Same stack, same failure family — worth a skim.

VPC Service Controls blocking Cloud Storage / API accesshigh

gsutil cp fails with 403 'denied by VPC Service Controls' or PERMISSION_DENIED

Google CloudAdvanced3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.