RAG
Intermediate 14 min read RAG

Retrieval Strategies

Fetching the right chunks, first

Retrieval connects the question to the right chunks: embedding search, keyword search, hybrid fusion, and reranking. The retriever determines what the LLM can even use.

retrievaltop-kembedding searchBM25hybridrerank
Prerequisites: Chunking Strategy
Before we start

What you will learn

  • Run embedding-only, keyword, and hybrid retrieval
  • Set top-k and rerank thresholds intelligently
  • Diagnose 'right doc, wrong chunk'
Introduction

What is this?

Retrieval takes an embedded question and returns candidate chunks via vector similarity, BM25-style keyword match, or both fused. A reranker then re-sorts candidates by finer-grained matching. The retriever is the funnel: garbage in means garbage prompts, so top-k, hybrid mix and reranking decide most of a RAG system's quality.

Motivation

Why do you need it?

  • The LLM can't use what never made the top-k — retrieval bounds the ceiling.
  • Hybrid fusion catches exact-string needs that embeddings miss.
  • Reranking fixes 'embeddings are good enough' by re-scoring semantically.
  • Top-k control is your latency + quality slider.
Think about it

Mental model

Imagine

Searching a library by category cards vs flipping to the right shelf

Think of

Retrieval

First you skim category shelves to shortlist (ANN top-N). A reference librarian (reranker) then checks a few shortlisted books closely for the exact passage. Wide net first, precise pick second.

Mechanics

How it works

  • Embed the query with the SAME model → vector search top-k.
  • Optional: keyword (BM25) pass over the same corpus.
  • Fuse lists (RRF/weighted) when both run.
  • Rerank the fused candidates with a stronger scoring model or cross-encoder.
  • Trim to the final top-k and pass to the prompt.

Retrieval stages.

Query
Vector top-N
BM25 top-N
Fuse
Rerank
top-k
Real world

In production

  • Policy engines hybrid-retrieve then cross-encoder-rerank for high-precision scope.
  • Code assistants use BM25 for symbol names plus vector for concept queries.
  • Simple support tickets ship embedding-only + fixed thresholds to start.
Guidelines

Best practices

  • Retrieve a wider top-N (e.g. 20) then rerank down to 4-8 for better precision.
  • Benchmark retriever variants on your labeled query set before tuning the prompt.
  • Return chunk scores and source ids — they power citations and not-found UX.
  • Cache embeddings for repeated queries.
Watch out

Common mistakes

  • Skipping rerank 'because embeddings are fine' on a noisy corpus.
  • Putting 20 giant chunks in the prompt and overflowing the context.
  • Using a different embedding model at query time than at ingest.
  • Hard-coding top-k with no notion of threshold.
Root cause analysis

Common issues

Error
High-similarity irrelevant hits
Root cause

Coarse embeddings or need for rerank

Fix

Add rerank, increase top-N before rerank, shrink k.

Error
Exact SKUs not retrieved
Root cause

No lexical leg

Fix

Add BM25/lexical or hybrid fusion.

When things break

Debugging tips

  • Right topic, wrong details → chunks are coarse; add rerank and tune top-k.
  • Missing exact identifiers → add a keyword/hybrid leg.
  • Retrieval infallible on demo, failing in prod → demo used canned queries; build a labeled set.
Interviews

Interview checkpoint

Practice

Hands-on lab

A/B Two Retrievers on Your Labels
35 min · intermediate
In progress
Objectives
  • Build two retrievers (vector + vector+lexical)
  • Score hit-rate@k on a labeled set
  • Pick the retriever by evidence

A/B Two Retrievers on Your Labels

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The role of reranking is to…

2

Retrieval quality caps the ceiling of RAG answers.

Go deeper

Official documentation

Retrieval in RAG (LangChain)
docs