Generative AI
Advanced 18 min read Generative AI

RAG in Production

RAG that survives contact with real users

Shipping RAG means measuring retrieval and groundedness, controlling latency (caching, re-rankers, hybrid search), handling chunk updates, and building evaluation sets that catch regressions before users do.

RAG evaluationgroundednesslatencyre-rankerhybrid searcheval setcaching
Prerequisites: RAG Vector Embeddings
Before we start

What you will learn

  • Build eval sets and metrics for retrieval quality
  • Measure groundedness/faithfulness of answers
  • Manage latency, caching and chunk-level freshness
Introduction

What is this?

Production RAG is a system with measurable components: retrieval precision (are the right chunks in the top-k?), groundedness (does the answer cite only the context?), and latency/cost (sub-second at your QPS). You need a labeled eval set of queries→expected chunks and queries→expected answers, plus pipelines that compute retrieval hit-rate@k and LLM-judged faithfulness on every corpus or prompt change. Freshness: chunk upsert/delete on doc change, plus versioned indexes for rollback.

Motivation

Why do you need it?

  • Quality user-reported bugs are too late; eval catches regressions pre-launch.
  • Latency budget decides architecture: re-rankers, cached chunks, embedding choice.
  • Freshness bugs (deleted docs still cited) erode trust and may breach compliance.
  • LLM-judged groundedness is the cheapest scalable hallucination detector.
Think about it

Mental model

Imagine

A support queue with QA gates between every handoff

Think of

Production RAG pipelines

Every retrieved document and every model answer passes a check — 'is this the right doc?' then 'is this answer faithful to the doc?' — like a QA stamp on each hand-off. Stamps are cheap; rework and angry users are not.

Mechanics

How it works

  • Eval set: 100-500 real queries with golden chunk-ids and golden brief answers.
  • Retrieval metric: hit-rate@k (gold chunk in top-k) and MRR; hybrid BM25+vector usually lifts hit-rate.
  • Re-rank: after ANN top-50, a cross-encoder scores the final top-5 — big quality-per-latency win.
  • Groundedness: have an LLM judge whether each claim in the answer is supported by the cited chunk.
  • Caching: dedupe embeddings and cache exact/prompt-keyed responses; LRU by recency.
  • Freshness: content hash → upsert/delete; run low-traffic index versions and hot-swap.

Production RAG with gates and measurement.

Query
ANN top-50
Re-ranker
LLM + prompt
Faithfulness check
Real world

In production

  • Always-on support/customer assistants with SLA and audit trails.
  • Compliance-sensitive Q&A where every answer must trace to a source doc.
  • High-QPS public search over docs where retrieval is the latency bottleneck.
Guidelines

Best practices

  • Automate the eval: run on CI at corpus and prompt changes; block on hit-rate/faithfulness regressions.
  • Instrument every span: chunk ids, scores, model, temperature, cache hit — the audit trail.
  • Set retrieval threshold: below it reply with a fallback rather than a hallucinated pass.
  • Cache aggressively (same-question repeat is common in support) and shard re-rankers by query type.
Watch out

Common mistakes

  • Eval-ing only answer 'niceness' and never retrieval precision — retrieval bugs hide.
  • No freshness story: edited docs keep being cited from the stale index.
  • Single-model single-prompt config with no regression hooks at all.
  • Ignoring tail latency — one slow re-rank fan-out collapses p99 under load.
Root cause analysis

Common issues

Error
hit-rate@k acceptable but answers still wrong
Root cause

Grounding gap — model ignores provided chunks

Fix

Harden the prompt, lower temperature, add a faithfulness gate that rewrites/declines non-grounded claims.

Error
p99 latency spikes under burst
Root cause

Re-ranker or decoding bottleneck fan-out

Fix

Cache common queries, cap top-k, batch re-rank, and stream the decode for first-token latency.

Error
Retrieved chunks cite deleted content
Root cause

Index not upsert/delete-synced with the source

Fix

Event-driven index reconciliation plus content-hash dedupe and a delete sweep on a schedule.

When things break

Debugging tips

  • Attach chunk-ids to every answer span so any user complaint maps to retrieved context.
  • Compute hit-rate@k per query-cluster; a dip in one cluster localizes the failing part of the corpus.
  • Shoot the same query through ANN vs hybrid vs re-ranked to see where the leak comes from.
  • Rotate a few real support tickets manually each week to ground automated metrics in reality.
Interviews

Interview checkpoint

Practice

Hands-on lab

Score a RAG Pipeline
35 min · advanced
In progress
Objectives
  • Compute hit-rate@k on a labeled mini-corpus
  • Implement a claim-level groundedness check
  • Build a latency budget sketch for QPS vs p95

Score a RAG Pipeline

35 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A faithfulness/groundedness check verifies:

2

RAG indexes must be refreshed when source documents change.

Go deeper

Official documentation

RAGAS eval framework
docs