AWS Runbooks

aws ec2 describe-instances shows no PublicIpAddress for a fresh instance

Instances in private subnets cannot reach package repos or APIs

aws s3 ls s3://<BUCKET>/ fails with AccessDenied on the instance

Freshly created role fails with AccessDenied/not authorized

All client types get 403 Forbidden on a bucket

Browser upload/download fails with AccessDenied immediately

Invoke returns 403 'The provided execution role does not have permissions'

Log group exists but log streams never appear

SSH to the instance times out or connection refused

Instance no longer in Running state — Stopping/Stopped or Terminated

df -h / shows 95-100% usage on the root device

aws ec2 describe-volumes shows State: attaching for minutes

ping/curl to the internet fails from an instance in a public subnet

Traffic fails in one direction only (e.g. inbound works, replies dropped)

App or pipeline fails with 'not authorized to perform sts:AssumeRole'

Load climbs but instance count stays flat

HTTP 503 Service Unavailable from the load balancer

One target shows unhealthy while others stay healthy

Apps error with 'too many connections' from the DB client

A production object/key range disappears after a cleanup script

CloudWatch log line: 'Task timed out after X seconds'

Alarm state shows INSUFFICIENT_DATA instead of OK or ALARM

Topic publish succeeds but subscribers never receive

top shows %st (steal) above 10-20% during an incident

aws ec2 describe-snapshots shows State: pending for hours

AttachVolume fails or volume stays in 'attaching' for encrypted volumes

Requests return 504 from the load balancer at steady rate

Readers return stale data relative to the primary

RDS status stuck in 'failover' or 'configuring-sync'

A role in account A cannot read a bucket owned by account B

Health checks report unhealthy while the endpoint responds fine

DLQ ApproximateNumberOfMessages keeps rising or flat at a ceiling

kubectl returns 'could not get token: AccessDeniedException' for a role

ec2 run-instances returns InsufficientInstanceCapacity

run-instances fails: There aren't enough IP addresses available in subnet

ASG scale-out spins up instances that immediately terminate

RDS status shows storage-full in the console

Each poisoned event processed twice (synchronous + async retry)

Primary region dies but DNS keeps answering with its address

New nodegroup instances launch but never appear in kubectl get nodes