Lesson 4 of 8 in Observability and incident response

Traces and OpenTelemetry: following one request

Advanced12 min

A trace is the recorded path of one request through a system, made of spans. A span is one timed unit of work, with a name, start and end time, and a link to its parent. OpenTelemetry is a vendor-neutral open standard and set of libraries for producing traces, metrics and logs.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Monitoring vs observability: metrics, logs, traces, Logs that help: structure, levels, correlation IDs

The problem this solves

A metric says checkout is slow. It does not say which of the six services involved is to blame. Logs from each service sit apart, and matching them by timestamp is guesswork.

A trace records, for one request, every step as a span with its duration and its parent. Laid out on a timeline, the slow step is the long bar.

For spans from different services to join into one trace, each service must know which trace it is part of. That is context propagation. OpenTelemetry standardises it so that different languages and vendors can cooperate.

How it works

A trace is a tree of spans
A trace is a tree of spansOne request creates a root span, and each sub-operation is a child span, so the tree shows what called what.parent ofparent ofparent ofcheckoutroot spancheck-stockchild spancharge-cardchild spandb-querygrandchild span
  • Control
  • Runs code
  • Data

One request creates a root span, and each sub-operation is a child span, so the tree shows what called what.

  1. checkout to check-stock: parent of
  2. checkout to charge-card: parent of
  3. charge-card to db-query: parent of
Context travels with the request
Context travels with the requestService A passes the trace ID to Service B in a request header, and both services send their spans to a collector that forwards them to a trace store.1 call + traceparent2 send spans3 send spans4 storeService Astarts the traceService Bcontinues the traceCollectorreceives spansTrace storeshows the timeline
  • Runs code
  • Control
  • Data

Service A passes the trace ID to Service B in a request header, and both services send their spans to a collector that forwards them to a trace store.

  1. Service A to Service B: 1 call + traceparent
  2. Service A to Collector: 2 send spans
  3. Service B to Collector: 3 send spans
  4. Collector to Trace store: 4 store

Every span in a trace carries the same trace ID, plus its own span ID and the span ID of its parent. The root span has no parent. The parent links are what let a tool draw the tree and a timeline.

When Service A calls Service B, the trace ID and A's current span ID travel in an HTTP header. The standard header is traceparent, defined by the W3C Trace Context specification. It looks like 00-<trace-id>-<parent-span-id>-<flags>, with the last field ending in 01 when the trace is sampled. Service B reads it and creates its spans as children of A's span.

OpenTelemetry provides the libraries that do this for you: an API your code calls, an SDK that records spans, and exporters that send them out. The OpenTelemetry Collector is an optional service that receives, processes and forwards telemetry to a backend. Because the format is standard, you can change the backend later without re-instrumenting your code.

Tracing every request can be costly, so systems often sample, keeping only a share of traces. A sampled trace is recorded in full, and an unsampled one is dropped.

Try it yourself

This script needs no backend. It uses the OpenTelemetry Python SDK with a console exporter that prints each finished span as JSON, and it simulates a second service by reading the propagation headers.

  1. 1
    Write a script that makes spans and passes context
    cd ~/obs-lab
    cat > trace_demo.py <<'EOF'
    from opentelemetry import trace
    from opentelemetry.propagate import extract, inject
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
    
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
    trace.set_tracer_provider(provider)
    tracer = trace.get_tracer("trace-demo")
    
    # Service A starts the trace and prepares an outgoing call
    headers = {}
    with tracer.start_as_current_span("checkout"):
        with tracer.start_as_current_span("check-stock"):
            pass
        with tracer.start_as_current_span("call-payments"):
            inject(headers)  # an HTTP client library normally does this for you
    print("headers sent:", headers)
    
    # Service B receives the headers and continues the same trace
    ctx = extract(headers)
    with tracer.start_as_current_span("charge-card", context=ctx):
        pass
    EOF
  2. 2
    Run it in a throwaway container
    docker run --rm -v "$PWD/trace_demo.py:/t.py:ro" python:3.12-slim \
      sh -c "pip install -q opentelemetry-sdk && python /t.py"

    Spans print as they finish, so inner spans appear before the span that contains them. The headers sent line shows a traceparent value.

  3. 3
    Compare the identifiers
    # in the printed JSON, compare these fields across the spans

    All four spans share one trace_id. check-stock and call-payments have checkout as their parent_id. charge-card, which ran after the context was extracted, has the span ID of call-payments as its parent, even though it was created in a different scope. The same ID also appears inside the traceparent header.

What breaks, and where to look

  • Each service shows its own separate trace of one span, and nothing links them.

    WHY IT HAPPENS
    The traceparent header is not reaching the next service. A proxy may drop it, a custom HTTP client may not be instrumented, or the call goes through a queue that does not carry headers.
    CHECK FIRST
    Log the incoming request headers in the second service and look for traceparent. If it is absent, find the hop that removes or never sets it.
  • The script runs and prints no spans, or the last spans are missing.

    WHY IT HAPPENS
    No tracer provider was registered before the tracer was used, or a batching processor still holds spans when the process exits.
    CHECK FIRST
    Call trace.set_tracer_provider(...) before creating tracers, and for batching processors call provider.shutdown() before exit so buffered spans are sent.
  • You look for the trace of a failed request and it does not exist.

    WHY IT HAPPENS
    Sampling dropped it. Only a share of traces are recorded when sampling is on.
    CHECK FIRST
    Check the sampler setting in your SDK or Collector, and use the request ID in logs to find the failure, since logs are usually not sampled.

When not to use this

  • A single service with one database often needs only metrics and good logs. Tracing pays off when a request crosses several services.
  • Tracing needs instrumentation in every service on the path. A trace that skips an uninstrumented service has a gap exactly where you may need it.

Check yourself

What is the relationship between a trace and a span?

A trace is the whole path of one request. It is made of spans, each one a timed unit of work linked to its parent.

What does context propagation do?

It passes the trace ID and parent span ID from one service to the next, usually in the traceparent header, so their spans join one trace.

Why use OpenTelemetry rather than a vendor's own library?

It is a shared standard for producing telemetry, so you can change the backend that stores it without rewriting your instrumentation.

Lesson 81 of 92 in the DevOps path

Search

Search cheat sheets and projects