AWS
Intermediate 14 min read AWS

AWS Monitoring and Observability

Metrics, logs, audits and traces on AWS

CloudWatch for metrics, alarms and logs; CloudTrail for who-did-what; X-Ray for traces. Together they answer 'is it up, and why is it slow or broken?'

CloudWatchCloudTrailX-Rayalarmslog groupsmetricstracing
Prerequisites: AWS Service Map Observability
Before we start

What you will learn

  • Use CloudWatch metrics + alarms for capacity signals
  • Explain CloudTrail's role in security and incident response
  • Trace a request with X-Ray across Lambda + API Gateway
In 60 seconds

CloudWatch collects metrics and logs, alarms and automates on them. CloudTrail records every API call that touched your account. X-Ray traces a request as it hops services. The three layers = health, audit, root cause.

Introduction

What is this?

CloudWatch is the monitoring layer: metrics (CPU, latency, custom), log groups for stdout/stderr, alarms that page SNS topics or trigger actions, and dashboards. CloudTrail logs management (and optionally data) events — who called what API when. X-Ray provides distributed tracing and service maps for Lambda, API Gateway and containerized apps. All three plug into SNS and EventBridge for automation.

Motivation

Why do you need it?

  • The cloud hides servers, so metrics are how you see inside — without them you're blind to correlated failures.
  • CloudTrail is legal/security 101: every AccessDenied and unauthorized change is evidence.
  • X-Ray turns 'the whole thing is slow' into a concrete 400ms hop on one Lambda.
  • Alarms with SNS are the cheapest alerting you own; spend there before adding an external tool.
Think about it

Mental model

Imagine

Instruments, security cameras and an operations room

Think of

CloudWatch, CloudTrail, X-Ray

Instruments (CloudWatch) measure heat, load and flow; cameras (CloudTrail) record who went where; the ops room (X-Ray) replays and connects the moments into one story.

Mechanics

How it works

  • CloudWatch collects AWS metrics automatically and custom metrics via the API/SDK or agent.
  • Alarms watch one or many metrics, transition state, and fire actions (SNS, Autoscaling, EC2 action).
  • Log groups store streams; a subscription filter can forward matching lines to Lambda/Kinesis.
  • CloudTrail trail → an S3 bucket + optional CloudWatch Logs; event history covers 90 days for free.
  • X-Ray instruments your code to emit segments; the service graph maps them into a single view.

AWS observability layers.

CloudWatch metrics
Alarms
CloudWatch Logs
CloudTrail
X-Ray
Real world

In production

  • A fleet of 40 EC2 instances alarms on CPU focus, autoscales up, and pages on health-check failure.
  • Security alarms on CloudTrail 'RootAccessKeyEvent' and 'UnauthorizedAttempt' events.
  • X-Ray traces an order Lambda through API Gateway, DynamoDB and SQS to find the outlier hop.
Guidelines

Best practices

  • Define SLOs and alarm on them, not on every metric; too many alarms robo-page.
  • Use structured JSON logs and one log group per service for greppable analysis.
  • Enable CloudTrail for all regions and protect the bucket (no public access, encryption).
  • Instrument custom business metrics (orders/min) — infrastructure metrics alone miss product outages.
Watch out

Common mistakes

  • Alarming everything honestly, then muting everything because it's too noisy.
  • Sending logs everywhere with no retention thinking — log group retention defaults eat storage.
  • Trusting a single region's metrics while the user is on another continent.
  • Forgetting metric resolution details: 1-min vs 5-min granularity changes your alarm latency.
Root cause analysis

Common issues

Error
Alarm pages at 3am every day for 2 minutes
Root cause

Daily batch spiking CPU; alarm too tight (single data point, no periods)

Fix

Set evaluation periods (e.g. 3 of 5) and add a perturb period to the metric.

Error
CloudTrail shows a deleted resource but no delete event from users
Root cause

A role/SCP removed it, or trail only covers one region

Fix

Enable org-level multi-region trails and match the event by sourceIP/AccessKeyId.

Error
Log stream grows unbounded, storage bill climbs
Root cause

No retention policy on the log group

Fix

Set retention (e.g. 14 days) via aws logs put-retention-policy.

When things break

Debugging tips

  • Slow response: dashboard CPU + latency side-by-side first, then X-Ray to find the hop.
  • Access oddity: go to CloudTrail event history and search by user/IP/action.
  • Lambda stuck: CloudWatch Logs show Init vs runtime; check memory log lines for OOM.
  • Alarm status=INSUFFICIENT_DATA usually means no metric at the resolution you expected.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which AWS service records the API calls made against your account for auditing?

2

CloudWatch metric alarms can trigger an automated scaling action.

Go deeper

Official documentation

AWS — CloudWatch user guide
docs
AWS — CloudTrail user guide
docs
AWS — X-Ray developer guide
docs