RAG
Advanced 14 min read RAG

Observing & Debugging RAG

From 'wrong answer' to root cause

When a RAG answer is wrong, the bug lives in one of three stages — corpus, retrieval, generation. Logging the right artifacts turns a mystery into a diagnosis.

RAG debuggingobservabilitytracesretrieval logsdiagnosis tree
Prerequisites: Evaluating RAG Systems
Before we start

What you will learn

  • Log retrieval artifacts per query
  • Run a three-way diagnosis tree
  • Inspect chunk-level traces
Introduction

What is this?

A wrong RAG answer is a symptom. The cause is usually corpus gaps, retrieval failure, or generation letting context drift. Observability records each stage's actual payloads — chunks retrieved, scores, final prompt, output — so any bad answer can be replayed and triaged to a stage instead of argued about.

Motivation

Why do you need it?

  • Debugging without artifacts is guessing with extra steps.
  • The three-stage split pinpoints optimal improvement automatically.
  • Replayed traces turn support complaints into reproducible tickets.
  • Regressions become watchable (stage metrics) rather than vibes.
Think about it

Mental model

Imagine

An airplane's flight recorder

Think of

RAG observability

After a rough landing, investigators replay the instrument tape: throttle (retrieval), altitude (context), engine (generation). They don't argue about who's guilty — they read data from each stage and fix precisely that system.

Mechanics

How it works

  • Attach a trace id per user question; prop it through ingestion and retrieval.
  • Log: query text, retrieved chunks (ids+scores), rerank output, final prompt tokens, output, finish reason.
  • Compute stage metrics in the trace: retrieval coverage, chunk scores, citation containment.
  • Stream traces to a dashboard with sampling for offline analysis.
  • Diagnosis tree: corpus? (content missing) → retrieval? (right chunks absent) → generation? (ignored context).

Trace anatomy.

Query
Retrieval
Rerank
Prompt
Output
Real world

In production

  • Support tickets escalate with a trace link — triage in minutes, not arguments.
  • Sample 1-in-100 traces for offline eval enrichment.
  • Stage dashboards alarm when retrieval coverage or groundedness ratio drops.
Guidelines

Best practices

  • Log chunk ids AND scores AND source ids — you need all three.
  • Capture the assembled prompt (or its diff) for debugging.
  • Keep traces sampled but deep; full detail is expensive at scale.
  • Build the diagnosis tree into the dashboard: click fail → see stage.
Watch out

Common mistakes

  • Logging only the final answer — the least useful artifact.
  • No trace id, so complaints can't be replayed.
  • Query-level stats without chunk-level views.
  • Debugging in prod without a sampled replay environment.
Root cause analysis

Common issues

Error
Repro requires asking the user again
Root cause

No trace id/artifacts persisted

Fix

Add trace ids and persist retrieval + prompt on failure paths.

Error
Traces are too big to inspect
Root cause

Full-detail logging at 100% sampling

Fix

Sample deeply (e.g. 2%, plus 100% on errors).

When things break

Debugging tips

  • Retrieved chunks present but answer wrong → generation stage; check prompts.
  • Right chunks absent → retrieval or corpus coverage; check scores/limits.
  • Retrieved noise dominates prompt → rerank/threshold too loose.
Interviews

Interview checkpoint

Practice

Hands-on lab

Replay and Triage a Bad RAG Answer
40 min · advanced
In progress
Objectives
  • Log stages under one trace id
  • Replay a trace to the diagnosis tree
  • Root-cause a failure to a stage

Replay and Triage a Bad RAG Answer

40 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The first thing to log per RAG query is…

2

A trace id turns a support complaint into a replayable ticket.

Go deeper

Official documentation

OpenLLMetry for RAG
docs