RAG
Intermediate 13 min read RAG

Corpus Embedding at Scale

The corpus is the knowledge

Sourcing a corpus, ingesting documents, and embedding them so the index never goes stale is half the RAG operation. Corpus matters: coverage and freshness beat clever algorithms.

corpusingestsource of truthfreshnessembeddingindex refresh
Prerequisites: Retrieval Strategies
Before we start

What you will learn

  • Audit corpus coverage against real questions
  • Design incremental, idempotent ingest
  • Keep the index fresh with the source of truth
Introduction

What is this?

Before retrieval there is a corpus: which documents, version of them, and how they flow into the index. RAG quality is capped by coverage ('is the answer in these docs?') and freshness ('is the answer current?'). Ingest should be incremental, idempotent and versioned so the index tracks the source of truth with bounded staleness.

Motivation

Why do you need it?

  • No retriever finds knowledge the corpus doesn't contain.
  • Stale docs answer yesterday's questions — often confidently wrong.
  • Ingest spans formats: markdown, PDFs, HTML, code — each needs a parser.
  • Idempotent upserts let refreshes run safely at any cadence.
Think about it

Mental model

Imagine

Restocking the reference shelf on a schedule

Think of

Corpus ingest

The knowledge shelf is only as good as the restock: new manuals in (fresh content), superseded ones out (deletion), and the shelf never quietly grows duplicates of last month's edition. Restock idempotently — running it twice safely is the whole point.

Mechanics

How it works

  • Enumerate sources from the system of record (CMS, repos, wikis).
  • Parse each format; normalize to text + metadata.
  • Chunk → embed → upsert with deterministic (source, revision, chunk) keys.
  • Delete/replace chunks whose source revision changed.
  • Monitor last-updated timestamps vs source counts; alert on lag.

Corpus duty cycle.

Sources
Parse
Refresh
Trash
Check
Real world

In production

  • Help-center RAG syncs from the CMS on publish events, triggering per-doc re-embed.
  • Repo-anything bots index pushes: updated files re-embed; deleted files remove chunks.
  • Weekly wiki crawls run idempotent upserts so re-runs are safe.
Guidelines

Best practices

  • Started from the true source of truth, never scraped exports.
  • Store (source, content-hash, chunk-index) with every vector.
  • Alert on freshness lag longer than a day.
  • Audit coverage quarterly — the corpus grows with the product or decays with it.
Watch out

Common mistakes

  • Scraping rendered pages instead of source formats.
  • No deletion path — retired docs linger in the index answering questions.
  • Re-embedding everything nightly instead of incremental diffing.
  • Ignoring permission/tenant metadata in the corpus.
Root cause analysis

Common issues

Error
Index grows unboundedly
Root cause

No delete/replace on revisions

Fix

Add superseded-revision cleanup keyed on the source revision.

Error
Fresh content never surfaces
Root cause

Ingest/webhook missed updates

Fix

Backstop with scheduled full-sync diffs and lag alerts.

When things break

Debugging tips

  • Questions with no corpus hit → coverage gap, not a retrieval bug.
  • Index size ≠ source count → deletes are missing; audit keys.
  • Docs updated but answers still old → freshness lag; check the ingest job's last run.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The primary cap on RAG knowledge is…

2

Retired documents should be removed from the index.

Go deeper

Official documentation

Ingestion patterns (LlamaIndex)
docs