Evaluating RAG Systems
Three scorecards, one dashboard
RAG evaluation is three metrics, not one: retrieval quality (recall), groundedness (are claims supported), and answer quality (is it useful). Measured together, they direct every optimization.
What you will learn
- Define retrieval, groundedness and answer metrics
- Build a golden set without leaking
- Use eval splits to direct fixes
What is this?
A RAG system can fail retrieving the right chunks, generating unsupported claims, or being unhelpful despite correct facts. Evaluation must measure all three separately: retrieval recall@k, groundedness (citation/containment rate), and answer quality (faithfulness + usefulness on context). Jointly they tell you WHICH stage to fix — the highest-leverage RAG decision.
Why do you need it?
- Retrieval metrics alone hide answer-quality problems and vice versa.
- A component scorecard prevents 'improved' rounds that made grounding worse.
- Golden sets build a regression net for the whole pipeline.
- Evidence-based tuning beats team intuition — especially as RAG gains complexity.
Mental model
A relay race with three timers
RAG evaluation
Which leg lost the race? Only per-leg timing tells you: retrieval (the baton handoff), grounding (not dropping facts), generation quality (not fumbling the finish). Timing one leg call it a 'race problem' hides where the actual fix lives.
How it works
- Golden set: (query, expected chunks) + high/medium/low reference answers.
- Retrieval eval: recall@k, MRR on expected chunks.
- Groundedness eval: claims ↔ chunk containment/citation validity.
- Answer eval: LLM-judge or human labels on faithfulness + helpfulness.
- Regression runs: same pipeline diff on each config change.
Three RAG scorecards.
In production
- Release gates: any config change must not regress the three scores.
- Fraction-of-error dashboard: '70% of bad answers trace to retrieval'.
- Live-to-eval loop: sample traffic into golden sets for continuous coverage.
Best practices
- Keep golden sets versioned, small, and representative (not just easy queries).
- Separate the eval for chunks (retrieval) from eval of final answers.
- Track a drift alert per stage — each has its own failure signature.
- Sample real user questions into goldens on a schedule.
Common mistakes
- Scoring only final answers — stuck not knowing which of three stages regressed.
- Evaluating on the training/tuning queries (leak).
- Asserting 'better answer' without grounding numbers worsening.
- Golden set stale while corpus grows.
Common issues
Eval flips while prod doesn't changeLLM-judge variance
Aggregate multiple runs, use deterministic references, and swap to human spot-checks periodically.
Golden queries too easySample bias
Pull real traffic, add adversarial/hard cases, keep distributions fresh.
Debugging tips
- Grounding score down, retrieval fine → generation stage: prompts, rerank feed, refusal logic.
- Retrieval down but answer decent → lucky top-k; fix chunks/retriever, don't celebrate.
- Answer quality 'better' but grounding worse → likely overfitting the judge or prompt tweaks.
Interview checkpoint
Hands-on lab
- Compute retrieval recall@k on golden chunks
- Score groundedness by containment
- Report a three-axis scorecard
Score a RAG Pipeline's Three Axes
40 min · intermediate
Quick quiz
Retrieval evaluation measures…
High answer quality implies the system is grounded.