Production Vector Serving
Running retrieval as a service
Serving vectors at scale is systems engineering: observability of recall, freshness of index, safe deployments and capacity planning. The store matters, but the wa around it makes retrieval trustworthy.
What you will learn
- Instrument recall and freshness
- Design rollout-safe index migrations
- Capacity-plan index memory and query load
What is this?
Production vector serving wraps the store with the disciplines you'd build for any datastore: recall monitoring, index freshness SLOs, deployment/migration playbooks, and capacity forecasting. The graph can rot, embeddings can drift, and migrations can mix spaces — each is a production incident if unmonitored.
Why do you need it?
- Recall degrades silently (graph rot, model swap, data skew) — and users feel it.
- Freshness: stale indexes answer yesterday's questions wrong.
- Vectors are migrations: changing a model invalidates the entire corpus.
- Capacity math (RAM, query QPS, ingest throughput) has only one chance to be right at launch.
Mental model
A city library with stats
Production vector serving
The library tracks how often each section gets the wrong book (recall), updates the catalog nightly (freshness), and re-shelves everything only during weekends (migration window). Nobody re-catalogs during a tsunami (rush hour) believing the results will hold.
How it works
- Recall probes: sample queries with known answers run vs a reference (brute-force or golden set).
- Freshness monitors: last-upsert timestamps vs source-of-truth change feed.
- Migrations: shadow index in the new space, eval against old, cut over with rollback.
- Capacity: model index memory, shard by tenant/region, plan for growth slope.
- Alerts: recall < threshold% or freshness lag > window page the team.
Vector serving pillars.
In production
- A RAG service runs 100 golden-query probes every hour comparing recall@5 vs a frozen snapshot.
- Doc-sourced indexes sync nightly from the CMS and alarm on stale chunks.
- Embedding-model upgrades fly as shadow indexes over weeks with automated cutover gates.
Best practices
- Define a recall@k SLO and measure it per tenant.
- Run freshness checks against the source of truth, not internal clocks.
- Script every embedding-model change as a rollout with rollback.
- Precompute capacity: dim × N graph memory + query headroom at p99.
Common mistakes
- Serving from an index whose data hasn't synced for days.
- Mixing old/new vectors during a model migration.
- Unmonitored recall: 'search just got worse' discovered by users, not alerts.
Common issues
Recall degrades over a monthGraph rot from inserts/deletes or embedding drift
Rebuild graph on schedule; re-tune M/ef; verify source vectors.
Migration cutover regressed recallNew index params/space not validated at scale
Re-run eval sweep; roll back while gaps close.
Debugging tips
- Recall dips pre-dated by an index rebuild → graph parameters or rebuild procedure changed.
- Freshness lag correlates with tenure spikes → ingest bottleneck, not the store.
- p99 latency up post-migration → index params or dimension changed; check recall too.
Interview checkpoint
Hands-on lab
- Build a tiny vector index from source text
- Run k-NN searches and inspect top-k
- Measure recall@k on a labeled query set
Where Recall Meets RAG
40 min · intermediate
Quick quiz
The best early-warning for silent retrieval decay is…
Embedding model swaps require no index migration.