mediumFoundationSymptom-first#cloudwatch#logs#agent#stream

CloudWatch Logs stream missing — agent not sending

Log group exists but log streams never appear

Likely source:configurationsecurity

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Log group exists but log streams never appear
  • Lambda/EC2 shows no logs in the console for hours
  • Filter patterns return zero matches that should exist

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Log group inventory

    aws logs describe-log-groups --log-group-name-prefix <PREFIX> --query 'logGroups[].logGroupName'

  • Streams in the group

    aws logs describe-log-streams --log-group-name <LOG-GROUP> --query 'logStreams[].logStreamName'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • EC2 instance runs no amazon-cloudwatch-agent
  • No recent agent logs on the host

Confirm before fixing

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["systemctl status amazon-cloudwatch-agent"]'
READ

Agent service state

Expected: inactive/dead or missing binary

Fix

aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["sudo amazon-linux-extras install collectd -y; sudo yum install -y amazon-cloudwatch-agent; sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s"]'
CHANGE

Install and start the agent with an EC2-managed config

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws logs describe-log-groups --log-group-name-prefix <PREFIX>
READ

Group present?

2
aws logs describe-log-streams --log-group-name <LOG-GROUP> --order-by LastEventTime --descending --max-items 5
READ

Streams present/recent?

3
aws ssm send-command --instance-ids <INSTANCE-ID> --document-name AWS-RunShellScript --parameters 'commands=["sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status"]'
READ

Agent status on the host

4
aws iam attach-role-policy --role-name <ROLE-NAME> --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
CHANGE

Fix permissions when that's the blocker

Prevention

  • Bake the CloudWatch agent and its config into your AMI/user-data.
  • Alarm on 'no log events in X hours' per critical group.
  • Give the agent role exactly CloudWatchAgentServerPolicy and nothing more.

Related runbooks

Same stack, same failure family — worth a skim.

CloudWatch alarm stuck INSUFFICIENT_DATAlow

Alarm state shows INSUFFICIENT_DATA instead of OK or ALARM

AWSIntermediate3 causes
Open runbook
Lambda execution role not authorized — invoke failshigh

Invoke returns 403 'The provided execution role does not have permissions'

AWSFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.