RAG
Intermediate 12 min read RAG

Reranking for Precision

Precise scoring over a shortlist

A retriever shortlists widely; a reranker scores precisely. Reranking (cross-encoder style) reliably lifts top-k precision and is the cheapest big win in many RAG stacks.

rerankcross-encoderprecisiontop-kscoring
Prerequisites: Retrieval Strategies
Before we start

What you will learn

  • Explain what a reranker scores that embeddings don't
  • Wire retrieve-wide-rerank-tight
  • Measure precision lift on a labeled set
Introduction

What is this?

Embedding similarity gives a coarse, pair-independent idea of relevance; a reranker sees the (query, chunk) pair together and scores actual relevance. Pipeline pattern: retrieve a generous top-N (10-30) with the cheap ANN, then rerank to the final small top-k with the precise scorer. It routinely adds 10-30% precision at modest latency cost on the shortlist.

Motivation

Why do you need it?

  • Cross-attention between query and doc catches interactions embeddings smooth over.
  • More precise feed → fewer hallucinations and better citations.
  • Reranking a shortlist is affordable: ~10-30 pairs, not 1M.
  • It corrects for noisy embedding spaces on tied-ish candidates.
Think about it

Mental model

Imagine

Two-pass hiring

Think of

Retrieve then rerank

Graphs of resumes get screened quickly into 20 finalists (retrieval). A panel then interviews those 20 — pairs each against the actual job (rerank) — and picks the true top 5, rather than trusting the screen for the final ranking.

Mechanics

How it works

  • ANN returns top-N candidates cheaply.
  • Each (query, chunk) pair runs through the reranker (cross-encoder or strong pairwise scorer).
  • Sort chunks by rerank score; drop below-threshold items.
  • Final top-k reaches the prompt; scores back citations.
  • Tune N-stage: bigger N costs latency; smaller N may drop the right chunk entirely.

Rerank stage.

Retrieve top-N
Pair-score
Sort+threshold
top-k
Real world

In production

  • Policy retrieval: rerank so the cited clause is the right one, not a sibling.
  • HAL-legal support: cheaper to rerank 30 and give 5 than to stuff 5 bad chunks.
  • Code assistants rerank function-level chunks for exact symbol matches.
Guidelines

Best practices

  • Default to retrieve-wide + rerank-tight (e.g. N=20 → k=5).
  • Benchmark rerank lift vs latency on your own labels.
  • Pass rerank scores to citations and keep the threshold for 'no good answer'.
  • Swap the embedding model and reranker independently — they're separate decisions.
Watch out

Common mistakes

  • Reranking N=5 — the decision is already made; the reranker sees only losers.
  • Treating rerank scores as calibrated probabilities (they aren't).
  • Skipping thresholds: reranking still must answer 'is ANY chunk good'.
Root cause analysis

Common issues

Error
Right chunk but wrong order into the prompt
Root cause

Reranker not applied or N too tight

Fix

Apply reranker over a wider N; verify ordering in logs.

Error
Rerank latency unacceptable
Root cause

N oversized or slow reranker

Fix

Trim N, batch pairs, or use a lighter reranker.

When things break

Debugging tips

  • Rerank doesn't improve → N too small or reranker/corpus mismatch; expand N, re-audit.
  • All rerank scores low → expected; use the score floor for not-found UX.
  • Latency up a lot → N too large for interactive; compress to 10-15.
Interviews

Interview checkpoint

Practice

Hands-on lab

Retrieve-Wide, Rerank-Tight
35 min · intermediate
In progress
Objectives
  • Retrieve top-5 then rerank down gradually
  • Prove precision improves with rerank scores
  • Pick N and k with labeled data

Retrieve-Wide, Rerank-Tight

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Rerankers score…

2

Reranker scores are calibrated doubt probabilities.

Go deeper

Official documentation

Cross-encoders (SBERT)
docs
Cohere rerank
docs