Observability
Intermediate 14 min read Observability

Tracing & APM

Rewind any request through every hop

Distributed tracing follows one request across services using propagated trace-ids and spans. APM layers on metrics/dashboards: service maps, error rates, and latency breakdowns that turn 'the checkout is slow' into 'the payments call is slow'.

tracespantrace-idOTelX-RayDatadogservice maproot cause
Prerequisites: Logging & Structured Logs
Before we start

What you will learn

  • Propagate trace context across services and queues
  • Interpret span trees with latency attribution
  • Use APM service maps for shotgun diagnosis
In 60 seconds

A trace = one request's tree of spans (client → LB → service → DB). Every hop adds a span carrying parent-id; the collector stitches by trace-id. APM visualizes as service maps + error/latency dashboards on top of trace data.

Introduction

What is this?

Tools: OpenTelemetry (SDK + collectors, vendor-neutral), AWS X-Ray, Azure Application Insights, GCP Cloud Trace, Datadog/New Relic APM. Protocol: W3C trace-context headers (traceparent), baggage. Span parts: name, start/end, status, attributes, links. Sampling: head (probabilistic at entry), tail (retain by result), rate-limited; OpenTelemetry parent-based sampler. Auto-instrumentation: language agents hook HTTP/DB frameworks (zero-code for common paths). APM surface: service map (topology + latency), golden signals per service, DB call breakdowns, error traces, slow-trace search. Correlation: attach trace-id to every log line for log↔trace join.

Motivation

Why do you need it?

  • Latency is almost never 'that box is slow' — it's a specific call in a chain.
  • Correlated logs + traces answer 80% of production bugs in minutes.
  • Sampling makes full-fidelity trace data affordable at traffic scale.
  • APM dashboards put the whole estate's shape on one reel.
Think about it

Mental model

Imagine

A courier's package timeline: pickup, sorting, each leg, with its own tracking number

Think of

Tracing

Every facility stamps its leg (span) attached to the same tracking number (trace-id). When a parcel is late, you see exactly which facility delayed it — not 'the courier is slow'.

Mechanics

How it works

  • Client creates trace-id + root span; sends W3C headers downstream.
  • Each service starts its own span with parent from the header; adds attributes.
  • Collector receives spans, groups by trace-id into complete tree.
  • Backend computes latency per span, percentile distributions, and service-linked graphs.
  • Dashboards + alerts hang off trace-derived series (APM).

Trace tree.

trace-id
API span
DB span
Queue span
Collector
APM view
Hands-on

Examples

OpenTelemetry-ish manual propagation

OpenTelemetry-ish manual propagation
// server middleware reads + activates context
const ctx = propagation.extract(global.active, req.headers);
const tracer = trace.getTracer('checkout');
const span = tracer.startSpan('handleOrder', { kind: 1 /*server*/ }, ctx);
// DB call gets child span; span ends on response
// log line: { trace_id: span.spanContext().traceId }

Explicit extraction + child spans; the collector needs nothing but the emitted spans.

Real world

In production

  • Auto-instrument the HTTP/DB layer fleet-wide via OTel agents.
  • Tail-sample errors at full fidelity + head-sample the rest to 5%.
  • APM alert: p99 trace root exceeded for the checkout path → page with trace links.
Guidelines

Best practices

  • Propagate trace context through EVERY hop — a dropped header kills the trace.
  • Keep span counts sane: don't over-instrument; one span per logical operation.
  • Attach trace-id in logs for the log↔trace join.
  • Sample deliberately (head + tail); never lose the errors.
  • Use semantic conventions so attributes stay consistent across services.
Watch out

Common mistakes

  • No propagation middleware → traces start and stop per service (orphan spans).
  • Sampling at 100% everywhere → bill melts for a lovely waterfall chart.
  • Logs without trace-id → you have the nice trace, but can't connect it to the event.
  • Instrumenting only 'interesting' endpoints → blind spots in the path you're trying to debug.
When things break

Debugging tips

  • A trace ends early → the backend service dropped the context (missing extract).
  • Big gap between consecutive spans → queue/network wait; check queue depth + message lag.
  • Trace shows one mega-service span → code didn't create child spans for DB/HTTP calls.
  • p99 spike traces can't reproduce → sample retains, so bias filtering by status/resource.
Interviews

Interview checkpoint

Practice

Hands-on lab

Trace One Request Across Two Handlers
35 min · intermediate
In progress
Objectives
  • Run OTel SDK with manual + automatic spans on two local services
  • Propagate context so the trace becomes a single tree
  • Inspect the tree in a local collector export

Trace One Request Across Two Handlers

35 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A span is:

2

The trace's most fragile step is:

Go deeper

Official documentation

OpenTelemetry traces
docs
AWS X-Ray
docs
W3C trace context
spec