Evaluation in Production
Scoring real traffic for silent regressions
Live evaluation samples production traffic and scores outputs with LLM judges, checks label or human feedback. It detects silent regressions after model swaps and prompt edits, turning observability into a closed loop.
What you will learn
- Sample production outputs for evaluation
- Apply judges and human feedback as labels
- Gate changes on live-eval deltas
What is this?
Production evaluation runs lightweight scoring on real traffic: sample traces, score answers with an LLM judge or explicit checks, tie scores back to model version, prompt version and segment. It turns 'answer seems off' into a measurable delta you can gate on.
Why do you need it?
- Offline evals miss what real users ask; production traffic is the honest distribution.
- Silent regressions (fluency up, correctness down) show up only with output scoring.
- Live deltas decide promotes/rollbacks faster than waiting for support tickets.
Mental model
Prod eval
A restaurant's mystery shopper
Every day a few random diners are quietly reviewed by a judge — so quality slips are caught this week, not in next quarter's complaints.
How it works
- Route a sampled subset of responses (say 5-15%) to an eval pipeline.
- Score with deterministic checks (schema, keyword, ref-match) plus an LLM judge on rubric.
- Tag each score with model/prompt/segment/hash so deltas are attributable.
- Expose eval trend per version; alert or auto-gate promotes/rollbacks.
Live eval feedback loop.
In production
- A support bot scores 10% of answers; a judge-rubric regression blocks the model promote.
- A summarizer flips prompt v3→v4 and watches fact-consistency score before widening.
- An agent platform checks each step's tool-call validity at production sample rate.
Best practices
- Sample deterministically — same trace can be re-judged after scoring changes.
- Score on a rubric aligned to product goals, not generic 'good/bad'.
- Pin scores to version hashes; compare like-for-like (same segment).
- Start small and monitor judge cost/latency so eval never slows the request path.
Common mistakes
- Scoring only errors — the silent garbage that matters is 200s.
- Comparing versions across changed segments (traffic mix drift).
- Judge overfitting to a rubric that rewards length over correctness.
Debugging tips
- Score drops with no deploy → segment or traffic mix changed; slice by intent.
- Judge flaky → move rubric thresholds or switch to a stronger judge for that dimension.
- Version A scores higher but users complain → rubrics miss the user-visible property; add it.
Interview checkpoint
Hands-on lab
- Sample one production-style request
- Score output with a checklist judge
- Compare two versions by score
Judge a Live Sample
35 min · intermediate
Quick quiz
Why sample production traffic for eval?
Scores only from offline golden tests cover production reality.