The Data Plane: Retrieval and State
What the model gets to read
The data plane feeds the model: vector stores for retrieval, caches, message history and caches of embeddings. Design for consistency and latency — index updates, staleness windows, and cache invalidation that stays fast under production load.
What you will learn
- Lay out the store behind the prompt (vectors, history, caches)
- Keep retrieval fast under write churn
- Bound staleness with index strategy
What is this?
The data plane is everything the model reads: the vector store behind retrieval, message history, caches (exact/semantic/prompt), and embeddings. Production realities — updates landing while reads run, index lag, cache eviction — are separate from pure algorithm design.
Why do you need it?
- Retrieval is usually the latency floor of a RAG request; its store must keep pace.
- History and caches must not silently serve stale or mixed-version content.
- Embedding millions of docs needs batching and versioned re-embedding.
Mental model
Data plane
A library's stacks and index
The librarian (retriever) finds books fast only if the catalog (index) is current and well-organized; a re-index happens while the stacks stay open (online update).
How it works
- Ingest: chunk, embed, write to the vector store in batches with a version tag.
- Retrieve: ANN search + rerank, return top-k to context assembly.
- State: message history in durable storage (Postgres/Redis) keyed by session.
- Caches: exact/prefix/semantic ahead of the model call.
Data plane for one request.
In production
- A support copilot searches a 10M-chunk store with p95 retrieval under 80ms.
- A shopping assistant keeps live inventory in a low-latency cache plus embedding search.
- A compliance app re-embeds only documents newer than a timestamp, not the whole corpus.
Best practices
- Store chunk text and metadata with every vector so answers can cite sources.
- Version documents so re-embedding rolls forward without tearing reads.
- Split the hot path: vector search by fixed knobs, results enriched from the source DB.
- Instrument retrieval latency and top-k relevance with the same trace as the model call.
Common mistakes
- Reading stale vectors after content changes — no re-embed queue or TTL.
- Storing vectors without metadata, so answers can't cite or filter.
- Synchronous embedding of every write at request time — writes block the path.
Debugging tips
- Bad retrieval → check embedding version of the stored chunk vs the query.
- Latency in retrieval → rerank-ahead vs store-ahead; move filters before the ANN.
- Answers citing gone content → the vector store holds old chunks; re-embed on change events.
Interview checkpoint
Quick quiz
The retrieval path's typical latency floor in RAG is?
Vectors should carry their source text and metadata.