NLP
Beginner 14 min read NLP

Text Preprocessing

From raw strings to a clean token stream

Raw text is messy. Tokenization, normalization, stopword removal, stem/lemmatization and vocabulary handling turn strings into the clean token sequences every NLP model consumes.

tokenizationstemminglemmatizationstopwordsnormalizationtext cleaning
Prerequisites: Python Pandas
Before we start

What you will learn

  • Tokenize and normalize text with the right rules per language/domain
  • Choose between stemming and lemmatization
  • Handle vocabularies, OOV tokens and unicode safely
Introduction

What is this?

Text preprocessing converts raw strings into the token sequences a model can learn from. The pipeline is order-sensitive: normalize case/unicode, tokenize (word, subword or character level), strip or retain stopwords by task, and reduce inflection via stemming or lemmatization. Every choice is a bias — aggressive cleaning destroys information (negation, sentiment), light cleaning leaves noise. The preprocessing should match the downstream model's needs, not an arbitrary checklist.

Motivation

Why do you need it?

  • Tweets, tickets and reviews are full of typo, casing, URL, emoji and spacing noise that kills naive feature work.
  • Vocabulary size explodes (memory + sparsity) if you don't normalize and prune.
  • Stem/lemmatize unifies surface forms so 'running' and 'run' count as one signal.
  • Done wrong, cleaning silently removes the signal you wanted to predict (e.g. 'not good').
Think about it

Mental model

Imagine

Washing, peeling and chopping vegetables before cooking

Think of

Text preprocessing

You peel (strip formatting), rinse (unicode/entities), chop to the right size (tokens), and often keep the skin when the recipe needs it — preprocessing is task-appropriate preparation, not a universal deep-clean that strips flavor.

Mechanics

How it works

  • Normalize: lower-case, NFC unicode fold, collapse whitespace, decode entities.
  • Tokenize: whitespace+punctuation splits (simple) or library tokenizers; for large vocab use subword (BPE) tokenization.
  • Filter: drop URLs/emails/mention tokens or keep placeholders — decide per task.
  • Infl. reduction: Porter stemmer (crude, fast) vs WordNet/lemmatizer (context-aware, slower).
  • Vocabulary: keep the top-k tokens, add <UNK> and <PAD>; encode indices and store as sparse/dense matrices.

The text-to-tokens pipeline.

Raw text
Normalize
Tokenize
Filter/reduce
Vocabulary
Hands-on

Examples

Preprocess a tweet with spaCy-style rules

Preprocess a tweet with spaCy-style rules
python3 -c "
import re
def clean(text: str) -> str:
    text = re.sub(r'http\S+', 'URL', text)
    text = text.lower()
    text = re.sub(r'[^a-z\s]', ' ', text)
    return re.sub(r'\s+', ' ', text).strip()
print(clean('Great deal!!! http://bit.ly/xyz (love it)'))
"

URLs become the placeholder, case is unified, punctuation collapses — tokens are 'great deal url love it'.

Real world

In production

  • Ticket-routing pipelines normalize and tokenize, but keep negations by NOT stripping all stopwords on the sentiment path.
  • Search indexers lemmatize queries and docs so 'purchased' matches 'purchase'.
  • LLM-era pipelines still preprocess: dedupe, de-emphasize boilerplate, normalize UTF-8 before embeddings.
Guidelines

Best practices

  • Write one idempotent preprocessing function per pipeline; reuse it at train and serve exactly.
  • Keep the language/domain explicit (hindi? dialect? code vs prose?).
  • Retain negation-aware tokens for opinion tasks (replace 'not' + adj with NOT_adj).
  • Log vocab stats — tokens, OOV rate, mean token length — with every experiment.
Watch out

Common mistakes

  • Lowercasing everything and destroying case-sensitive entities (USB, GDPR, McDonald's).
  • Stripping all stopwords on tasks where 'not', 'but' and 'because' carry the meaning.
  • Different preprocessing online vs offline (serve got the cleaned text; training didn't).
  • Letting <UNK> dominate: >2% OOV on a fixed vocab means re-tok, grow vocab or switch to subword.
Root cause analysis

Common issues

Error
Model treats 'not good' and 'good' identically
Root cause

Stopword removal deleted 'not'

Fix

Keep negations — either skip stopword removal or turn 'not X' into a negation-feature token.

Error
Huge OOV rate at inference
Root cause

Live vocabulary different from training

Fix

Rebuild vocab over combined corpora (or use subword tokenization) and add <UNK> handling.

When things break

Debugging tips

  • Print 50 tokenized samples of each class before firing up a model — eyeball beats assumption.
  • If accuracy stalls at a ceiling, check preprocessing isn't removing the predictive token (negation, price symbols).
  • Compare vocab coverage on train vs serve corpora; drift inflates OOV.
  • Hash-sensitive: reproduce the same token stream from the same raw string in every environment.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Tokenizer With Metrics You Trust
30 min · beginner
In progress
Objectives
  • Implement a reproducible normalization + tokenization pipeline
  • Measure vocab size, OOV rate and sentence-length distribution
  • Compare stopword-on vs stopword-off on a sentiment-flavored sample

Build a Tokenizer With Metrics You Trust

30 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

For sentiment analysis, removing all stopwords is risky because:

2

Aggressive text cleaning always improves model performance.

Go deeper

Official documentation

spaCy tokenization
docs
NLTK text processing docs
docs