Data Handling
Beginner 13 min read Data Handling

Document Processing for AI

From messy files to clean text

Most enterprise knowledge lives in PDFs, HTML and docx. Extracting clean text — pages, structure, tables, metadata — is the first step of every RAG and document-GPT pipeline, and it shapes retrieval quality.

document parsingPDFOCRHTMLmarkdownmetadataRAG
Before we start

What you will learn

  • Extract text and structure from PDF, HTML and office documents
  • Decide when OCR and layout analysis are necessary
  • Keep metadata so chunks stay linked to their source
Introduction

What is this?

Document processing turns files — PDFs, HTML pages, docx, scans — into clean text plus structure plus metadata that downstream pipelines can embed and retrieve. It is deceptively hard: PDFs store text in unpredictable orders, tables break naive text extractors, columns confuse page readers, and images need OCR. Getting it right is a prerequisite for any quality RAG system.

Motivation

Why do you need it?

  • Garbage in, garbage out: a bot answering from mis-extracted PDF text hallucinates details that were never there.
  • Layout-aware extraction preserves marks, tables and reading order that naive text tools destroy.
  • Metadata (source URL, page, section, timestamps) is what makes citations and traceability possible.
  • Format-specific quirks (columnar PDF, complex tables in Excel, HTML nav noise) all need different handling.
Think about it

Mental model

Imagine

Repacking a moving box

Think of

Document extraction

Moving boxes arrive as mixed items; you unpack, sort and label so the kitchen knows what's cooking. Extraction unpacks a file into labeled, ordered text — until then, the 'content' isn't usable in any quick way.

Mechanics

How it works

  • Detect the format and pick an extractor: pdfplumber/PyMuPDF for PDFs, BeautifulSoup for HTML, python-docx for Word.
  • For scanned documents, OCR (tesseract, cloud OCR) converts pixels to text, optionally with layout analysis.
  • Read the layout: merge columns, preserve headings, keep tables as structured rows where possible.
  • Normalize: unicode fixes, whitespace cleanup, and keep an ordered text stream plus a metadata sidecar.
  • Stream documents to the chunker with their provenance attached so every chunk can cite its source.

Document → text → metadata-rich stream.

File
Format extractor
OCR
Normalize
Text + metadata
Hands-on

Examples

Extract markdown from HTML

Extract markdown from HTML
from bs4 import BeautifulSoup
 
html = "<html><body><h1>Pricing</h1><p>Plans start at <b>$9</b>.</p></body></html>"
soup = BeautifulSoup(html, "html.parser")
 
text = " ".join(soup.get_text(" ", strip=True).split())
print(text)  # "Pricing Plans start at $9."

Simple HTML stripping with BeautifulSoup preserves the essentials; production pipelines handle nav, scripts and tables with more care.

Real world

In production

  • A legal-docs RAG uses layout-aware PDF extraction plus OCR for archived scans, tagging each chunk with case, page and section.
  • A support-bot crawls a help site, strips HTML noise, and embeds only the main-content region, cutting hallucinated answers.
  • A financial pipeline parses table-heavy filings with structured extraction so cells become queryable data instead of gluing text.
Guidelines

Best practices

  • Sample real documents first — synthetic PDFs hide the layout traps production has.
  • Keep original source and extraction metadata with every chunk.
  • Handle unicode the moment it's extracted, not at embed time.
  • Re-extract when the source changes; never cache stale text into the index silently.
Watch out

Common mistakes

  • Using a single naive 'get_text()' on a columnar PDF and losing the reading order.
  • OCR-ing text-based PDFs (slow, error-prone) when the text layer exists.
  • Dropping tables into one undifferentiated blob and expecting retrieval to make sense of them.
  • No sampling/QA on the extracted corpus, so garbled text trains the bot silently.
Root cause analysis

Common issues

Error
PDF extraction returns text in the wrong order
Root cause

Multi-column layout read left-to-right across whole page

Fix

Use a layout-aware extractor (PyMuPDF blocks, pdfplumber) or OCR with analysis mode.

Error
OCR output is full of typos
Root cause

Low scan resolution or complex fonts

Fix

Upscale images before OCR and validate a sample against ground truth.

Error
Tables come out as gibberish
Root cause

Table structure lost in naive text extraction

Fix

Use structured table detection/parsing and serialize cells as rows.

When things break

Debugging tips

  • Render a page image and overlay extracted text — misalignment means your chunker is reading wrong columns.
  • Compare character counts raw-vs-extracted to find pages where extraction silently dropped content.
  • If a section never shows up in search, check whether extraction merged it into a neighbor's span.
Interviews

Interview checkpoint

Practice

Hands-on lab

Walk a Document Through Parse-Clean-Chunk
25 min · beginner
In progress
Objectives
  • Parse a raw markdown page into structured units
  • Normalize text and split a long page along its structure
  • Emit a chunk registry that maps chunks back to source headings

Walk a Document Through Parse-Clean-Chunk

25 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

When do you need OCR for a PDF?

2

Extraction metadata (source, page, section) is essential for traceable RAG answers.

Go deeper

Official documentation

PyMuPDF documentation
docs
pdfplumber docs
docs