Observability
Beginner 10 min read Observability

Observability for LLM Apps

Beyond up/down: why is the answer bad?

LLM apps fail interestingly: garbage output with a 200 status. Observability means tracing each generation, its inputs, cost, latency and downstream outcome so you can see why an answer was wrong — not just that the API was up.

observabilitytracesmetricsLLM appdebugging
Prerequisites: Serving Gateways
Before we start

What you will learn

  • Explain why LLM apps need output-level visibility
  • Name the four observability pillars for AI
  • Structure a request-scoped trace
Introduction

What is this?

Observability captures what happened to every request end to end: the prompt, the model, parameters, tokens used, per-stage latency, cost and the final output — correlated under one trace ID. It upgrades debugging from 'did it error' to 'why did it answer wrong and cost that much'.

Motivation

Why do you need it?

  • A model can return plausible garbage with HTTP 200 — status codes rarely explain AI failures.
  • Cost and quality issues are invisible without per-call token and output telemetry.
  • Regressions follow deploys (model swaps, prompt tweaks) and need per-version comparison.
Think about it

Mental model

Imagine

Observability

Think of

A black-box flight recorder

The plane landed (service up), but the recorder shows the interesting stuff — throttle, altitude and decisions at every second.

Mechanics

How it works

  • Every request gets a trace ID propagated through gateway, retries and streams.
  • That trace carries prompt, model, params, TTFT, tokens, cost and final text.
  • Metrics aggregate: latency percentiles, error rate, token spend, cache hits by model.
  • Sampling keeps the highest-value traces (errors, edge cases, outliers).

One trace, many signals.

Input
Call
Output
Outcome
Hands-on

Examples

What a trace should store

What a trace should store
trace = {
  "id": "tr_9f2", "version": "v7", "model": "fast-model",
  "prompt_hash": "ab12", "ttft_ms": 340, "latency_ms": 1800,
  "total_tokens": 512, "cost_usd": 0.0041,
  "output": "the full answer", "outcome": "accepted",
}

Replayable and comparable: same inputs → different output is visible as a model/config change.

Real world

In production

  • A platform team compares traces across model versions to explain regressions.
  • An agent pipeline logs each tool call with its own span inside the parent trace.
  • Cost dashboards slice spend by tenant, model and endpoint.
Guidelines

Best practices

  • Trace from before the model call to after the user outcome.
  • Include the version of prompt and config — not just timestamps.
  • Redact PII before storing full text.
  • Sample error and slow traces at 100%, success traces at ~10%.
Watch out

Common mistakes

  • Logging only errors — garbage answers aren't errors and vanish.
  • No trace ID end-to-end, so a bad answer can't be rejoined to its inputs.
  • Storing full prompts without redaction.
When things break

Debugging tips

  • 'Works in staging': compare trace fields (model, params, context) between environments.
  • Sudden cost growth: trace-level token sums beat aggregate guessing.
  • Weird outputs: replay the exact prompt hash + params from a stored trace.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Trace with Standard Attributes
30 min · beginner
In progress
Objectives
  • Model a request as a span tree
  • Attach gen-AI attributes to a span
  • Export a synthetic trace to console

Build a Trace with Standard Attributes

30 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The most AI-specific thing observability must capture is?

2

A 200 status means the LLM call went fine.

Go deeper

Official documentation

OpenTelemetry
docs
Observability for LLMs
spec