Distributed Tracing

• In a microservice architecture, a slow response has many possible causes across many services.
• Traces reveal the actual call graph and per-span latency, replacing guesses about where time went.
• Context propagation links logs, metrics and traces for the same request through a shared trace ID.
• Sampling keeps the cost and storage of tracing predictable at scale.
• OpenTelemetry gives one standard for instrumentation, so one SDK covers HTTP, gRPC, databases and queues.

Dropping breadcrumbs at each step is logging — you know where you were, but not the route you took or how long each leg lasted. A GPS track log is a trace: one continuous record of the whole journey with timing on every segment. Traces show the map; logs are the notes scribbled at each landmark.

• The first service to receive a request generates a 128-bit trace ID and starts a root span for the entry operation.
• The tracing SDK injects context into outbound calls using the traceparent header, which carries trace ID, parent span ID and a sampled flag.
• Downstream services extract the header, create child spans with the same trace ID, and propagate the context to their own dependencies.
• Each span records its name, start and end timestamps, attributes, status and events; an exporter batches them to a collector or backend.
• A tracing backend such as Jaeger or Grafana Tempo reconstructs spans sharing a trace ID into the request's tree for visualization.

• The platform SDK auto-instruments HTTP, gRPC and database clients, so every service emits spans without per-team work.
• On-call opens the trace of a slow checkout and sees in the waterfall exactly which downstream service added the 4 seconds.
• Tail-based sampling captures rare error paths that head-based sampling would have dropped at the entrance.
• Trace IDs written into every log line let a support ticket number resolve straight to a call graph.
• The trace backend feeds RED metrics — rate, errors and duration — per endpoint, replacing hand-written latency gauges.

• Propagate context across every boundary: HTTP headers, message queues, jobs and database calls.
• Keep span names low-cardinality and static, e.g. GET /checkout, never embedding user IDs.
• Sample intentionally: head-based sampling for high-volume traffic, tail-based sampling to keep rare errors.
• Add stable attributes like service.version, http.route and error.type to spans for richer queries.
• Log the trace ID in every service so correlation survives even when a trace is sampled out.

• Not propagating the header, so every service starts a new root trace and the story fragments.
• Adding high-cardinality attributes like user IDs to every span, inflating the backend's storage.
• Sampling 100% head-based and burning the budget on healthy traffic instead of failures.
• Naming spans with dynamic values, which destroys the backend's ability to aggregate by operation.
• Instrumenting only some services, so the trace is full of gaps and hard to reason about.

• Sort spans by duration; the widest gap between child spans indicates waiting on a dependency.
• Fetch the same trace a few times and compare durations to distinguish a slow path from an intermittent spike.
• Filter spans with error status or error.type attributes to jump straight to the failing call.
• Grep logs for the trace ID to see application-level context around each span boundary.
• Check the sampling policy when a trace is missing — it may have been deliberately dropped, not lost.