Observability
Beginner 12 min read Observability

Observability Foundations

From 'is it down' to 'which request path broke it'

Observability is the ability to answer 'why is it doing this' from the outside — three pillars: metrics (numbers), logs (events with text), traces (request journeys). Monitoring tells you it's broken; observability helps you understand it.

observabilitymetricslogstracesSLOpillarstelemetry
Prerequisites: Cloud Monitoring
Before we start

What you will learn

  • Distinguish metrics, logs and traces and their use cases
  • Set cardinality-aware instrumentation (RED/USE)
  • Drive SLIs/SLOs from the three pillars
In 60 seconds

Metrics: cheap, high-cardinality-dimensioned numbers (latency, errors, saturation). Logs: structured events with context — good for forensics, expensive at scale. Traces: span trees showing request flow across services. Operationally: USE for resources, RED for services.

Introduction

What is this?

Pillars: metrics (Prometheus/CloudWatch/Azure Monitor/Stackdriver; counters, gauges, histograms; label dimensions), logs (structured JSON via CloudWatch/Log Analytics/Loki), traces (OpenTelemetry spans+, X-Ray/App Insights/Google traceparent). Concepts: cardinality (10 dimensions × 100 values each = explosion), sampling, retention/aggregation tiers, SLI (measure), SLO (target), error budgets. Cloud-native telemetry: CloudWatch Metrics + X-Ray, Azure Monitor + App Insights + Log Analytics, Cloud Monitoring + Trace + Logging; OpenTelemetry as the provider-agnostic emitter. RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) are the two starter dashboards.

Motivation

Why do you need it?

  • You can't fix what you can't see — 'works in prod' is not a metric.
  • The three pillars are complementary: numbers point at the layer, logs prove the event, traces show the path.
  • SLOs turn 'we feel it's fine' into numbers the whole org agrees on.
  • Well-instrumented apps make decade-long incidents 10-minute diffs.
Think about it

Mental model

Imagine

A car dash: gauges (metrics), event recorder (logs), and service logs from parts (traces)

Think of

Observability pillars

Speed/revs tell you something's wrong (metrics); the recorder shows the exact event (logs); tracing the part failures shows which component caused the stall (trace).

Mechanics

How it works

  • Apps report metrics at intervals (counters/gauges/histograms) with dimensions.
  • Log messages arrive as structured JSON with a request/trace id.
  • Traces propagate a trace-id/span across every hop; collectors aggregate.
  • Observability backends store/query each pillar with its own tradeoffs.
  • Dashboards/alerts key off metrics while on-call drills into logs + traces.

Three pillars → three answers.

Service
Metrics
Logs
Traces
Dashboards / alerts
Hands-on

Examples

RED for an HTTP service

RED for an HTTP service
rate(http_requests_total{job="api"}[5m])      # Rate
sum(rate(http_requests_total{status=~"5.."}[5m])) /
  sum(rate(http_requests_total[5m]))            # Errors
histogram_quantile(0.99, http_request_duration_seconds) # Duration (p99)

Three queries = one service dashboard; the whole fleet visible on a single paint.

Real world

In production

  • OpenTelemetry SDK in every service → one collector → one backend.
  • SLO dashboards for core services with error-budget burn-rate alerts.
  • Structured request logs correlated to traces by trace-id (bug triage in 10 minutes).
Guidelines

Best practices

  • Standardize on OpenTelemetry for export; one vendor-agnostic path.
  • Mind cardinality: keep label sets small and bounded.
  • Instrument at all layers: client, LB, service, DB, queue.
  • Define SLIs before dashboards so numbers draw charts with intent.
  • Store hot data at high-res, cold at reduced retention (cost sanity).
Watch out

Common mistakes

  • Metrics with unbounded dimensions (user_id labels) → storage explosion.
  • Logs without trace-ids → you have the event, not the cause.
  • Dashboards that are works of art with zero alert coverage.
  • SLO defined on a metric no one verified (the missing SLI first).
When things break

Debugging tips

  • p99 vs mean: always look at percentiles for latency, the mean lies.
  • Metrics empty but logs noisy → exporter/collector/batching config, then code.
  • Trace shows a foreign service gap → missing propagation header at that boundary.
  • Storage cost creep → review sampling + retention per pillar quarterly.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which pillar is best for p99 latency alerting?

2

An SLI is:

Go deeper

Official documentation

OpenTelemetry docs
docs
Google SRE book (monitoring)
docs
Prometheus docs
docs