Text Preprocessing for LLMs
What your model actually reads
Normalization, cleaning and segmentation decide what your embeddings and prompts actually see. Small choices — unicode normalization, newline handling, section detection — measurably change retrieval and generation quality.
What you will learn
- Normalize unicode, whitespace and casing consistently
- Remove boilerplate without losing meaning
- Segment text into sections that respect semantic boundaries
What is this?
Text preprocessing is the deterministic layer between the raw document and the chunk/embed/prompt stage: unicode normalization (é vs e+combining), whitespace and newline handling, boilerplate removal (nav, footers, cookie banners), bullet/list structure, and section segmentation. LLMs are token-blind to most of this, but embedding quality and chunk coherence depend heavily on it.
Why do you need it?
- Splitting at a real section boundary beats splitting at an arbitrary character count, every time.
- Two differently-encoded spellings of the same product create duplicate embeddings and split retrieval results.
- Boilerplate can dominate a small chunk and drown real content.
- Consistency matters more than cleverness: identical text should always preprocess identically.
Mental model
Polishing raw gemstones
Text preprocessing
You don't jump from rock to ring — you clean off the dirt, split the stone along a real crystal face, and only then cut facets. Preprocessing is that cleaning and splitting; skip it and the gem (your chunks) comes out cloudy.
How it works
- Normalize unicode first (NFC), then fix whitespace: collapse runs, standardize newlines, trim.
- Detect and strip boilerplate regions (headers, footers, navigation) with heuristics or trained callers.
- Preserve list/bullet structure so retrieval can pull a single item, not a glued paragraph.
- Segment by semantic boundaries (headings, blank paragraphs, layout signals) before any size-based chunking.
- Keep the pipeline deterministic and idempotent so re-processing is stable.
The preprocessing chain.
In production
- A catalog bot normalizes product names across imported feeds so embedding duplicate detection finds real duplicates.
- A docs-knowledge base strips nav and banner HTML before chunking, so content chunks stop being half-navigation.
- A support index splits at `##` headings instead of raw character counts, making retrieval hit whole how-tos.
Best practices
- Normalize unicode to NFC early and everywhere.
- Process deterministically and store the cleaned text — never clean twice in different places.
- Remove boilerplate conservatively; over-aggressive stripping destroys context.
- Segment before chunking: structural boundaries beat fixed sizes for coherence.
Common mistakes
- Cleaning in production differently than in eval, so results don't transfer.
- Destroying list structure by joining all lines — retrieval can't find one product then.
- Normalizing after embedding, guaranteeing cache and embedding mismatches.
- Using a regex to 'remove numbers' that also removes invoice and SKU content.
Common issues
Same entity appears as two different chunksInconsistent normalization (é vs e) across sources
Apply NFC unicode normalization uniformly at ingestion.
Boilerplate dominates small chunksNav/footer text still in the segment
Strip boilerplate before chunking and QA the cleaned corpus.
Lists lose their individual itemsNewlines collapsed flattening bullets into one block
Preserve bullet structure so items segment independently.
Debugging tips
- Debug against a single known document: render cleaned output next to raw and eyeball the diff.
- If retrieval misses content, search for it in cleaned text — it may have been stripped or merged.
- Two chunks that should be identical but aren't — check unicode/case normalization consistency.
Interview checkpoint
Quick quiz
What is the safest ordering of preprocessing steps?
Cleaning must be identical in production and evaluation.