highAdvancedUSE#ec2#cpu#steal#performance

High CPU steal on EC2 — noisy neighbor / oversubscription

top shows %st (steal) above 10-20% during an incident

Likely source:performanceavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • top shows %st (steal) above 10-20% during an incident
  • App latency rises while instance CPU utilization (ACPUCredit) looks low
  • Requests pile up despite the app being 'idle' by its own metrics

Steal is invisible in CPUUtilization because that metric only counts instance-level vCPU time. Always check %st on the host for slow-isn't-busy mysteries, and never run latency-critical stateful workloads on t-series at sustained load.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Live steal percentage

    aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["top -bn1 | head -15"]'

  • CPUUtilization vs expectations

    aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --dimensions Name=InstanceId,Value=<INSTANCE-ID> --start-time <ISO> --end-time <ISO> --period 300 --statistics Average

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • %st is elevated while %us/%sy of the instance is low
  • Same app running on another AZ has normal %st

Confirm before fixing

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["vmstat 1 5 | awk '{print $14}'"]'
READ

Watch steal tick up over several samples

Fix

aws ec2 stop-instances --instance-ids <INSTANCE-ID> && aws ec2 start-instances --instance-ids <INSTANCE-ID>
CHANGE

Stop/start to move off the contended host (loses public IP unless EIP)

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["vmstat 1 5"]'
READ

Confirm steal is the dominant wait

2
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUCreditBalance --dimensions Name=InstanceId,Value=<INSTANCE-ID> --start-time $(date -u -d '-6 hours' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period 3600 --statistics Minimum
READ

Rule out credit exhaustion on burstable types

3
aws ec2 stop-instances --instance-ids <INSTANCE-ID>
CHANGE

Plan a move off the contended host

4
aws ec2 start-instances --instance-ids <INSTANCE-ID>
CHANGE

Restart on fresh hardware

Prevention

  • Alert on host-level 'st' from the CloudWatch agent / node exporter, not just CPUUtilization.
  • Use dedicated-core or latest-gen instance types for latency-sensitive services.
  • Prefer t3/t4g unlimited or right-sized m/c instances for steady CPU load.
  • Add spread/placement-group awareness so replicas land on distinct hosts.

Related runbooks

Same stack, same failure family — worth a skim.

EC2 instance unreachable — status checks failingcritical

SSH to the instance times out or connection refused

AWSIntermediate3 causes
Open runbook
High CPU throttling — containers starved by CPU limitsmedium

High response latency with low CPU saturation on the node

KubernetesAdvancedUSE2 causes
Open runbook
Auto Scaling group not scaling out under loadhigh

Load climbs but instance count stays flat

AWSIntermediate4 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.