Generative AI
Intermediate 17 min read Generative AI

Retrieval-Augmented Generation (RAG)

Grounding answers in documents you control

RAG grounds LLM answers in retrieved documents: embed your knowledge base, fetch the most relevant chunks for a query, and feed them to the model with the question. It fixes hallucinations and staleness without retraining.

RAGretrievalvector searchchunkingembeddinggroundinghybrid search
Prerequisites: Inference & Decoding Vector Embeddings
Before we start

What you will learn

  • Describe the full RAG pipeline: index, retrieve, augment, generate
  • Choose chunking, embedding and hybrid search for a domain
  • Identify the failure points: re-ranks, missing context, retrieval quality
Introduction

What is this?

RAG splits a corpus into chunks, embeds each, and stores them in a vector index. At query time the top-k chunks are retrieved (semantic similarity; often hybrid keyword+vector) and inserted into the prompt as context so the model answers from them. The LLM never invents the facts — the retrieved passages are the ground truth. RAG updates instantly when new documents arrive, no retraining involved.

Motivation

Why do you need it?

  • Fresh knowledge: re-index new docs and the model 'knows' them today, not at next finetune.
  • Grounding kills hallucinations: the answer must cite the supplied context.
  • Compliance: you control exactly which documents are in-scope, and the model 'only' sees them.
  • Cheap and fast: no GPU fine-tuning, just an embedding step and a search index.
Think about it

Mental model

Imagine

A librarian who reads the book to you, pointing at the page

Think of

RAG

You ask a question; the librarian indexes the library, finds the three most relevant pages, and reads you the answer from those pages — page numbers cited. They don't recite the whole library from memory, and when a new book arrives they just shelve and index it.

Mechanics

How it works

  • Index: chunk documents (by sections, ~200-1000 tokens), embed each chunk into a vector.
  • Store: push vectors + metadata (source, date, chunk id) into a vector DB with ANN index.
  • Retrieve: embed the query, k-NN search, get top-k passages (optionally hybrid: BM25 + vector, fused).
  • Augment: build the prompt = [system] + retrieved context blocks + query.
  • Generate: the LLM answers citing the chunks; a check step verifies answers against context.

Index once, retrieve + generate per query.

Docs
Vector index
Query
Context
Grounded answer
Real world

In production

  • Enterprise knowledge assistants over wikis, manuals, contracts.
  • Support-ticket agents with policy grounding and citation links.
  • Codebase Q&A, legal/billing document lookup at scale.
Guidelines

Best practices

  • Chunk by semantic boundaries (headings/paragraphs) not fixed char counts when possible.
  • Index metadata and include it in the prompt so the model can cite sources and dates.
  • Hybrid search (BM25 + vector) beats pure vector on proper nouns and part-numbers.
  • Set a no-match path: when the top score is below threshold, say you don't know — don't force an answer.
Watch out

Common mistakes

  • Chunking that splits a concept mid-sentence; retrieval then returns half-truths.
  • Embedding the whole doc as one vector — retrieval granularity is terrible.
  • No score threshold, so irrelevant chunks still reach the model and leak noise.
  • Testing only with exact-phrase queries; evaluation must cover paraphrase and out-of-scope queries.
Root cause analysis

Common issues

Error
Good corpus but bad answers
Root cause

Right chunks not retrieved or prompt ignores them

Fix

Inspect top-k chunks for the failing query; tune chunking/hybrid search and instruct the model to use context only.

Error
Retrieval misses paraphrased queries
Root cause

Vector-only search and thin embeddings

Fix

Add hybrid BM25, try stronger embedding models, or query expansion.

Error
Stale answers after doc updates
Root cause

Index not refreshed on edit/delete

Fix

Upsert-and-delete on doc events; add a freshness timestamp to chunks and surface it in citations.

When things break

Debugging tips

  • Inspect retrieved chunks manually for 20 sample queries BEFORE judging answer quality.
  • Log chunk ids and scores with every answer — grounding checks become reproducible.
  • Difference between answer and context? Test retrieval alone: does the right chunk survive the top-k?
  • A/B chunking sizes on your own eval set; 200-token is not universal.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Mini RAG Pipeline
40 min · intermediate
In progress
Objectives
  • Chunk and embed a small document set
  • Run cosine-similarity retrieval for a query
  • Assemble a grounded prompt from the top-k chunks

Build a Mini RAG Pipeline

40 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The main reason RAG reduces hallucinations is:

2

Adding a new document to the corpus requires retraining the LLM.

Go deeper

Official documentation

Retrieval augmented generation (blog)
docs