Evaluating Embeddings
Scoring meaning with mathematics
Embeddings are judged on your task: retrieval hit rate, dedup F1, clustering coherence. Building small labeled eval sets and threshold sweeps turns 'feels right' into numbers you can ship.
What you will learn
- Build a small labeled eval set
- Score retrieval and dedup tasks
- Run a threshold sweep correctly
What is this?
True embedding quality shows up in task metrics: retrieval hit-rate@k over labeled queries, dedup precision/recall on known pairs, clustering coherence on annotated buckets. Every embedding-model swap and threshold change is judged on these small, owned eval sets — the scoreboard that keeps 'the embeddings feel good' honest.
Why do you need it?
- BLEU-style text metrics are irrelevant; task metrics are the truth.
- A 3-model benchmark beats a year of gut feelings.
- Threshold sweeps need a validation set separate from the test set — leak silently ruins them.
- Regressions after upgrades are caught pre-rollout, not in production logs.
Mental model
A driving school's exam track
Embedding eval sets
You don't judge a driver by how fast they go in a straight line (spec sheet) — you put them on a fixed track with known hazards (labeled queries/pairs) and score cornering and braking (hit rate, F1). The track stays the same so every candidate is comparable.
How it works
- Build labeled data: (query, relevant doc) pairs and (near-duplicate / distinct) pairs.
- Retrieval eval: embed queries, search, measure hit-rate@k or MRR@k.
- Dedup eval: precision/recall/F1 at your operating threshold.
- Threshold sweep: step through similarity values on validation, pick argmax F1, confirm on test.
- Track all of it in one script so model swaps are reproducible.
The eval loop.
In production
- CI runs the embedding eval on every model/chunk config change.
- RAG teams gate retrieval model upgrades on eval lift (or no regression).
- Threshold reviews cite hit-rate/F1 numbers rather than screenshots.
Best practices
- Keep eval sets small but representative and versioned.
- Separate validation (tune thresholds) from test (final verdict).
- Re-run the eval before AND after enabling any embedding change.
- Record model, metric, normalization in the eval output header.
Common mistakes
- Tuning thresholds on the test set (overfit the eval).
- Sampling eval queries from the training corpus (leak).
- Comparing models at different thresholds.
- Judging dedup F1 with unlabeled pairs handed to you by 'a reviewer'.
Common issues
Eval disagrees with prod experienceEval set doesn't match real queries
Enrich eval with real traffic samples (safely, after logging).
Model upgrade eval regressionNew space unaligned with your conventions
Evaluate on task metrics and pin the version if worse — upgrades are opt-in, not automatic.
Debugging tips
- Eval score drops but demo looks fine → demo eyeballs aren't eval; trust the script's subset.
- hit-rate@k too low → chunking/query formulation, not just model.
- F1 jumps at two thresholds → bimodal clusters; re-inspect labels before trusting.
Interview checkpoint
Hands-on lab
- Embed a small corpus and query it
- Cluster + threshold to find near-duplicates
- Compute a dedup F1 with labeled pairs
Embed a Corpus and Dedup It
40 min · intermediate
Quick quiz
A good retrieval eval metric for embeddings is…
Thresholds should be tuned on the test set for best results.