lowIntermediateSymptom-first#cloudwatch#alarm#metrics#insufficient-data

CloudWatch alarm stuck INSUFFICIENT_DATA

Alarm state shows INSUFFICIENT_DATA instead of OK or ALARM

Likely source:configuration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Alarm state shows INSUFFICIENT_DATA instead of OK or ALARM
  • Page does not fire during a real incident
  • Alarm worked, then went gray after a metric rename

INSUFFICIENT_DATA alarms do not page by default — that's how to silently lose an outage. The state-reason text is the fastest diagnostic, and 'treat missing data as notBreaching' turns noise into meaningful greens.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Alarm configuration

    aws cloudwatch describe-alarms --alarm-names <ALARM-NAME> --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason,Metric:MetricName,Namespace:Namespace,Dimensions:Dimensions,Period:Period}'

  • Raw metric existence

    aws cloudwatch list-metrics --namespace <NAMESPACE> --metric-name <METRIC-NAME> --dimensions Name=<DIM-KEY>,Value=<DIM-VALUE>

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • list-metrics shows no matching metric
  • Instance or service emitting it is off

Confirm before fixing

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["systemctl status amazon-cloudwatch-agent"]'
READ

CloudWatch agent state on the host

Fix

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["sudo systemctl restart amazon-cloudwatch-agent"]'
CHANGE

Restart the agent / fix the app emission

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws cloudwatch describe-alarms --alarm-names <ALARM-NAME> --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason,Namespace:Namespace,Metric:MetricName,Dimensions:Dimensions}'
READ

Read target + reason

2
aws cloudwatch list-metrics --namespace <NAMESPACE> --metric-name <METRIC-NAME>
READ

Does the metric exist?

3
aws cloudwatch get-metric-statistics --namespace <NAMESPACE> --metric-name <METRIC-NAME> --dimensions Name=<DIM-KEY>,Value=<DIM-VALUE> --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period <PERIOD> --statistics Average
READ

Do datapoints exist?

4
aws cloudwatch put-metric-alarm --alarm-name <ALARM-NAME> --namespace <CORRECT-NS> --metric-name <CORRECT-NAME> --dimensions <DIM-KEY>=<DIM-VALUE> --period <PERIOD> --statistic Average --evaluation-periods <N> --treat-missing-data notBreaching --comparison-operator GreaterThanThreshold --threshold <T> --alarm-actions <SNS-ARN>
CHANGE

Rebuild the alarm against the real emission

Prevention

  • Set treat-missing-data explicitly on every alarm so gray is a rare, meaningful signal.
  • Dashboards: add a 'last datapoint age' tile to spot silent producers early.
  • Alert on CloudWatch agent health itself when using custom metrics.

Related runbooks

Same stack, same failure family — worth a skim.

CloudWatch Logs stream missing — agent not sendingmedium

Log group exists but log streams never appear

AWSFoundation3 causes
Open runbook
Route53 health check false negativeshigh

Health checks report unhealthy while the endpoint responds fine

AWSAdvanced3 causes
Open runbook
EC2 instance unreachable — status checks failingcritical

SSH to the instance times out or connection refused

AWSIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.