RAG
Intermediate 13 min read RAG

Chunking Strategy

Cutting the corpus into answer-sized pieces

Chunking decides how the corpus is cut for retrieval — and it decides whether answers exist in one retrievable piece. Structure-aware chunking with overlap beats raw character splits.

chunkingchunk sizeoverlapstructure-awaresemantic chunking
Prerequisites: RAG in One View
Before we start

What you will learn

  • Chunk structurally (headings, paragraphs, code blocks)
  • Tune size/overlap with evidence
  • Diagnose retrieval misses as chunking problems
Introduction

What is this?

Chunking splits source documents into retrievable units. Too big: chunks blur topics and blow context. Too small: answers split across chunks that top-k can't reassemble. Structure-aware splitting (by headings/paragraphs/code blocks) with a little overlap usually beats fixed-char cuts because chunks align with where answers actually live.

Motivation

Why do you need it?

  • Retrieval quality is bounded by chunk quality — no retriever finds an answer split across chunks.
  • Char-only splitting severs tables, code and headings mid-thought.
  • Similar, composable chunks embed more coherently than mixed-topic slabs.
  • Overlap only where needed patches boundary-splitting without bloating the index.
Think about it

Mental model

Imagine

Cutting a manual into tabbed sections

Think of

Chunking

You cut along chapter boundaries, not every 600 characters — a 500-char cut halfway through 'Troubleshooting' would lose half the fix. Overlap just a bit of the previous chapter's tail so the ending still has context.

Mechanics

How it works

  • Parse structure (markdown headings, HTML sections, PDF meta) before splitting.
  • Split units preserving boundaries; keep code fences and tables intact.
  • Add overlap (8-15% or a sentence or two) to temper boundary loss.
  • Optionally merge short units so chunks have coherent minimum size.
  • Evaluate: does the answer live mostly inside single retrieved chunks?

From document to chunks.

Document
Parse structure
Split
Chunks
Index
Hands-on

Examples

Structure-aware splitter sketch

Structure-aware splitter sketch
import re
 
MARKDOWN = "## Install\nRun pip install acme\n\n## Troubleshoot\nCheck the logs at /var/log/acme."
 
def split_markdown(text: str):
    parts = re.split(r"\n(?=## )", text)
    # keep headings attached, add a small overlap of the previous tail
    chunks = []
    prev_tail = ""
    for p in parts:
        body = prev_tail + p
        chunks.append(body)
        prev_tail = " ".join(p.split()[-25:])
    return chunks
 
for c in split_markdown(MARKDOWN):
    print("CHUNK:", c.split("\n")[0], "| words:", len(c.split()))

Doc structure decides the cut; a tail overlap keeps boundary context.

Real world

In production

  • Docs sites chunk by page+section so each retrieval can point at one section.
  • Repos chunk by file+function so answers stay function-scoped.
  • Internal wikis use tables as atomic units; they embed poorly when split.
Guidelines

Best practices

  • Chunk by structure first, then tune size/overlap by retrieval eval.
  • Keep code fences, tables and list blocks intact.
  • Store the section path (doc, heading) with each chunk for citation.
  • Re-chunk when retrieval eval says answers are 'split across chunks'.
Watch out

Common mistakes

  • Splitting on exact character counts into generic uniform slabs.
  • Zero overlap, so sentences at boundaries lose their beginning.
  • Chunking inside code blocks or table rows.
  • Forgetting chunk changes invalidate the existing index (re-embed needed).
Root cause analysis

Common issues

Error
Key facts straddle chunk boundaries
Root cause

Chunking ignores structure or overlap

Fix

Split at headings; add sentence-level overlap; consider two-level chunking.

Error
Retrieval recalls unhelpfully big slabs
Root cause

Chunks oversized

Fix

Sub-split while preserving structure; evaluate recall@k again.

When things break

Debugging tips

  • Answers 'almost but missing the tail' → overlap too small or cut mid-sentence.
  • Answers merge two unrelated topics → chunks too big; split at headings.
  • Retrieval finds the doc but wrong section → chunk granularity vs question scale.
Interviews

Interview checkpoint

Practice

Hands-on lab

Chunk a Doc and Prove Recall Improves
35 min · intermediate
In progress
Objectives
  • Compare char-split vs structure-aware chunking
  • Show boundary-splitting breaks retrieval
  • Tune overlap and measure single-chunk recall

Chunk a Doc and Prove Recall Improves

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Best practice is to split chunks…

2

Changing chunking does not change the index.

Go deeper

Official documentation

Recursive text splitter (LangChain)
docs