Embeddings
Intermediate 13 min read Embeddings

Embedding Pipelines

Turning docs into a living index

An embedding pipeline takes corpus → clean → chunk → embed → index and query → re-embed. Reliability, idempotency and monitoring turn 'embeddings' from a demo into a service.

embedding pipelineingestchunkingidempotentre-embedpipeline
Prerequisites: Embedding Models
Before we start

What you will learn

  • Design an ingest pipeline end to end
  • Make ingestion idempotent and resumable
  • Plan for model and content changes
Introduction

What is this?

An embedding pipeline is a repeatable process: normalize source documents, split into chunks, compute embeddings, upsert into a vector store, and rebuild/refresh as content or the model changes. What separates a demo from a service is idempotency (re-running is safe), versioning (models gut-tracked), and monitoring (coverage, drift, orphan vectors).

Motivation

Why do you need it?

  • Embeddings are stale the moment content changes — pipelines must refresh.
  • Idempotent upsert means re-runs don't duplicate or corrupt.
  • Chunking quality decides retrieval quality far more than model choice often does.
  • Model upgrades are migrations, not background chores.
Think about it

Mental model

Imagine

Updating a library's card catalog

Think of

Embedding pipelines

New book (content) arrives → catalog a card (embed), file it (index), and cross-check old cards (refresh). Rebinding the collection (model upgrade) means re-cataloging everything, not just the new arrivals. A good librarian does this without ever losing a card or misfiling two for one title.

Mechanics

How it works

  • Normalize: strip format noise, mark sections, handle attachments.
  • Chunk: split with overlap by structure (headings, paragraphs) — never mid-sentence.
  • Embed: batch and cache; attach source, chunk-id, model-version metadata.
  • Upsert: replace by (source, chunk-id) — idempotent keys.
  • Query: embed the question with the SAME model and search the same space.

The vector lifecycle.

Sources
Normalize+chunk
Embed
Upsert
Query
Real world

In production

  • Doc-index refresh jobs re-run nightly with upsert keys (source, rev, chunk).
  • Support copilots re-embed FAQ updates the same day they edit the source.
  • Model upgrade flight: dual-write new-space to a shadow index, cut over on eval improvement.
Guidelines

Best practices

  • Give every chunk a deterministic ID for idempotent upserts.
  • Track embedding model + chunking config as a version in every record.
  • Run a nightly coverage check: index size vs source counts.
  • Maintain a rerunnable re-embed script for model upgrades.
Watch out

Common mistakes

  • Full rebuilds on every change instead of incremental upserts.
  • Ignoring chunk boundaries (mid-table splits destroy meaning).
  • Querying with a different model than indexed.
  • Orphan/duplicate vectors after deletions.
Root cause analysis

Common issues

Error
New content never appears in search
Root cause

Ingest pipeline errored on that batch

Fix

Expose per-batch failures, retry with idempotent keys, alert on staleness.

Error
Duplicates after rerun
Root cause

Non-deterministic upsert keys

Fix

Key on (source, content-hash, chunk-index).

Error
Model swap broke thresholds
Root cause

Old vectors still in the index

Fix

Re-embed whole corpus; keep thresholds in config tied to model+metric.

When things break

Debugging tips

  • Search misses new docs → ingest job failed silently; check batch errors + latest-upsert time.
  • duplicate-heavy results → missing deletes; audit upsert keys.
  • Score distribution changed suddenly → embedding model or chunking config drifted.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The safest upsert key for a chunk is…

2

Queries must embed with the same model family as the index.

Go deeper

Official documentation

Embedding pipelines (LangChain style)
docs