criticalIntermediateSymptom-first#ec2#status-checks#ssh#unreachable

EC2 instance unreachable — status checks failing

SSH to the instance times out or connection refused

Likely source:availabilityperformance

Focused incident mode — strips away everything but this runbook.

Symptoms

  • SSH to the instance times out or connection refused
  • AWS console shows 1/2 or 2/2 status checks failed
  • App health checks against the instance are down

Remember the reboot/stop-start distinction: stop-start moves the instance to a new physical host, which is the only remedy for host-level impairment. It also changes the public IP unless an EIP is attached.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Status check state

    aws ec2 describe-instance-status --instance-ids <INSTANCE-ID>

  • Instance state and reason

    aws ec2 describe-instances --instance-ids <INSTANCE-ID> --query 'Reservations[0].Instances[0].{State:State,Reason:StateTransitionReason}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • InstanceStatus check failed while SystemStatus is ok
  • Console output shows a hang or panic after boot

Confirm before fixing

aws ec2 get-console-output --instance-id <INSTANCE-ID> --latest --query 'Output' | base64 -d | tail -50
READ

Inspect the tail of the boot log

Expected: Rebooting, restarting system, or a panic trace

Fix

aws ec2 reboot-instances --instance-ids <INSTANCE-ID>
CHANGE

Reboot the instance from the API

Expected: rebooting

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ec2 describe-instance-status --instance-ids <INSTANCE-ID>
READ

Establish which status check (system vs instance) failed

2
aws ec2 get-console-output --instance-id <INSTANCE-ID> --latest --query 'Output' | base64 -d | tail -50
READ

Read the boot/serial log

3
aws ec2 reboot-instances --instance-ids <INSTANCE-ID>
CHANGE

First recovery move for a hung OS

4
aws ec2 stop-instances --instance-ids <INSTANCE-ID> && aws ec2 start-instances --instance-ids <INSTANCE-ID>
CHANGE

Stop/start for a fresh host if reboot does not clear it

Prevention

  • CloudWatch alarm on StatusCheckFailed_Metric for every production instance.
  • Keep the SSM agent and a known-good AMI so you can debug without SSH.
  • Use an Elastic IP or a load balancer so public addressing survives stop/start.

Related runbooks

Same stack, same failure family — worth a skim.

EC2 instance stopped or terminated unexpectedlycritical

Instance no longer in Running state — Stopping/Stopped or Terminated

AWSIntermediate3 causes
Open runbook
New instance has no public IP addresshigh

aws ec2 describe-instances shows no PublicIpAddress for a fresh instance

AWSFoundation2 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.