Observability
Intermediate 12 min read Observability

Tracing LLM Calls

Spans that tell a story

An LLM trace is a request-scoped record: parent span for the whole call, child spans for prompt assembly, the model call itself, tool invocations and streaming. Correlated ids let you replay and compare any request that matters.

tracespanOpenTelemetryLLM spanscorrelation
Prerequisites: Observability Overview
Before we start

What you will learn

  • Model an LLM call as parent + child spans
  • Attach generative-AI attributes to spans
  • Correlate traces to metrics and logs
Introduction

What is this?

Tracing structures per-request telemetry: one root span for the request, child spans for each stage — building the prompt, calling the model, running tools, finalizing. Gen-AI convention attributes (gen_ai.request.model, gen_ai.usage.input_tokens) ride on the model span so tools can render, search and compare them.

Motivation

Why do you need it?

  • Nested stages (RAG, agents) fail in different spans; a flat log can't show which hop leaked.
  • Attributes let you query 'all traces using model X with temp 0.8 and slow TTFT'.
  • Traces correlate: one id links the span tree, aggregate metrics and any logs.
Think about it

Mental model

Imagine

Tracing

Think of

A shipping manifest's packing list

The manifest (trace) references boxes (spans); each box lists contents (attributes) so you can find what moved where, and when.

Mechanics

How it works

  • Start a root span at request entry; propagate trace/span ids via headers.
  • Create child spans per stage; measure their duration and status.
  • Set standard attributes: model, provider, tokens, temperature, tool calls, latency.
  • Export via OTLP to a collector/backend (Langfuse, Arize, Datadog, OTel).

Span tree for one chat call.

Root
Prompt build
Model call
Tool run
Hands-on

Examples

Minimal OTel-style span

Minimal OTel-style span
span = client.start_span("chat", attributes={
    "gen_ai.request.model": "fast-model",
    "gen_ai.usage.input_tokens": 812,
    "gen_ai.usage.output_tokens": 204,
    "gen_ai.system": "openai",
})
result = call(span)          # propagate trace headers
span.set_output(result)      # store the text
span.end()

Standard attributes keep traces queryable across providers and tools.

Real world

In production

  • An agent platform shows the full loop: plan → tool → observe → next plan as one trace.
  • A search team finds which retriever stage added 600ms by drilling one trace's spans.
  • Langfuse/Arize UIs render prompt, tokens and verdict alongside the spans.
Guidelines

Best practices

  • Propagate trace/span ids through async and streaming boundaries.
  • Set a consistent attribute schema (semconv) across providers.
  • Store truncated output plus a link to redacted full text.
  • Export in batches; never block the request path on telemetry.
Watch out

Common mistakes

  • Starting the root span after retries — you lose the retry context.
  • Not closing spans → broken durations and lost relationships.
  • Credentials or full prompts in attribute values (PII leak).
When things break

Debugging tips

  • One span dwarfs all → that stage is the budget hole; zoom into its own child spans.
  • Missing child spans → propagation headers weren't forwarded into the worker.
  • Trace differs between envs → compare attribute values (model, params, retrieval).
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A parent span for a chat request most often contains what?

2

Traces are useful only when something errors.

Go deeper

Official documentation

OpenTelemetry genai semantics
spec
Langfuse
docs