criticalIntermediateSymptom-first#ec2#spot#credits#stop#termination

EC2 instance stopped or terminated unexpectedly

Instance no longer in Running state — Stopping/Stopped or Terminated

Likely source:availability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Instance no longer in Running state — Stopping/Stopped or Terminated
  • App traffic stops immediately, no shift to a replacement
  • CloudWatch AWS/EC2 StatusCheckFailed metric goes flat

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • State and transition reason

    aws ec2 describe-instances --instance-ids <INSTANCE-ID> --query 'Reservations[0].Instances[0].{State:State,Reason:StateTransitionReason,AZ:Placement.AvailabilityZone}'

  • Spot interruption signal (if spot)

    aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name InstanceInterruptionWarning --dimensions Name=InstanceId,Value=<INSTANCE-ID>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • StateTransitionReason == Client.UserInitiated
  • A scheduled-stop job or operator action ran around the StopTime

Confirm before fixing

aws ec2 describe-instances --instance-ids <INSTANCE-ID> --query 'Reservations[0].Instances[0].{Reason:StateTransitionReason,StopTime:StopTime}'
READ

Confirm the reason and time

Expected: Client.UserInitiated with a StopTime

Fix

aws ec2 start-instances --instance-ids <INSTANCE-ID>
CHANGE

Restart the instance if the stop was unintended

Expected: RUNNING

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ec2 describe-instances --instance-ids <INSTANCE-ID> --query 'Reservations[0].Instances[0].{State:State,Reason:StateTransitionReason}'
READ

Identify who/what stopped it

2
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<INSTANCE-ID> --lookup-attributes AttributeKey=EventName,AttributeValue=StopInstances
READ

Trace the API call through CloudTrail (if enabled)

3
aws ec2 start-instances --instance-ids <INSTANCE-ID>
CHANGE

Restore the instance for unintended stops

4
aws ec2 modify-instance-attribute --instance-id <INSTANCE-ID> --disable-api-termination --value true
CHANGE

Lock it down with termination protection

Prevention

  • Enable CloudTrail for EC2 management events so every stop/terminate is attributable.
  • Enable disableApiTermination on critical singleton instances.
  • Run spot workloads behind an ASG with a diversified allocation strategy.
  • Alert on InstanceStateChange events in EventBridge and tag instances for their schedule.

Related runbooks

Same stack, same failure family — worth a skim.

EC2 instance unreachable — status checks failingcritical

SSH to the instance times out or connection refused

AWSIntermediate3 causes
Open runbook
InsufficientInstanceCapacity — unable to launch or scalehigh

ec2 run-instances returns InsufficientInstanceCapacity

AWSProduction2 causes
Open runbook
Auto Scaling group not scaling out under loadhigh

Load climbs but instance count stays flat

AWSIntermediate4 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.