NLP
Intermediate 15 min read NLP

Bag of Words and TF-IDF

From tokens to vectors without any learning

Bag-of-words turns documents into count vectors; TF-IDF reweights by rarity so distinctive terms dominate. Both remain the cheap, robust feature basis for classical NLP.

bag of wordsTF-IDFdocument-term matrixsparsityn-gramsCountVectorizer
Prerequisites: Text Preprocessing
Before we start

What you will learn

  • Build document-term count matrices
  • Explain TF-IDF weighting and its properties
  • Use n-grams and limits (sparsity, vocab size) in practice
Introduction

What is this?

Bag-of-words (BoW) represents each document as a vector over vocabulary — count of each term. TF-IDF multiplies term frequency by inverse document frequency — log(N/df) — so terms common in few documents (jargon) get weight while stopword-like ubiquitous tokens fade. The result is a sparse, interpretable feature matrix that linear models, SVMs and quick baselines can run on instantly.

Motivation

Why do you need it?

  • Zero parameters: you can build a working classifier in minutes from raw text.
  • TF-IDF naturally down-weights noise ('the', 'report', 'please') and rewards distinctive terms.
  • Sparse algebra (scipy) keeps the matrix cheap even with 100k-token vocabs.
  • It's the honest baseline that embedding models must beat — and often the one that wins on small data.
Think about it

Mental model

Imagine

A filing clerk's index of keyword hits per folder

Think of

BoW/TF-IDF

The clerk counts how often 'invoice' appears per folder (BoW) but also notes that 'invoice' appears in almost every folder while 'reconciliation' appears in few — TF-IDF makes the rare-but-specific word matter more.

Mechanics

How it works

  • Fit the vocabulary over the corpus, optionally ≤ max_features and ≥ min_df.
  • Count term occurrences per doc into a sparse matrix (n_docs × n_vocab).
  • Apply IDF: w = tf × log((1+N)/(1+df)) + 1, then L2-normalize rows by default.
  • Add n-grams (word pairs etc.) to capture short-phrase semantics at schema cost.
  • Train any linear/regularized classifier on the matrix; interpret via coefficient weights.

Documents → counts → TF-IDF → model.

Docs
Counts
TF-IDF
Model
Interpret
Hands-on

Examples

TF-IDF by hand

TF-IDF by hand
python3 -c "
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ['invoice reconciliation monthly', 'monthly invoice', 'reconciliation audit']
v = TfidfVectorizer().fit(corpus)
print(sorted(v.vocabulary_.items()))
print(v.transform(corpus).toarray().round(3))
" 2>/dev/null || echo "sklearn missing"

'audit' appears in one doc so its IDF weight is large; 'monthly' in two docs gets a smaller weight boosted only where present.

Real world

In production

  • Support-ticket routing uses a TF-IDF+linear classifier retrained nightly — explainable and milliseconds-fast.
  • Search relevance re-ranks candidates with a TF-IDF/BM25 score as a cheap first filter ahead of embeddings.
  • Regulatory text mining uses word-coefficient weights to cite 'why' a doc was flagged.
Guidelines

Best practices

  • Analyse with sublinear_tf to damp hyper-frequent terms inside a doc.
  • Trim the vocab with min_df/max_df to desk noise and duplicate boilerplate.
  • Cache the fitted vectorizer and reuse it at inference — re-fit on live text silently skews weights.
  • For phrase-heavy domains, enable char n-grams or word n-grams where order matters.
Watch out

Common mistakes

  • Fitting the vectorizer on the whole corpus before splitting (IDF leaks test statistics).
  • Feeding raw counts into a model that wants scaled inputs — most need the L2-normalized TF-IDF.
  • Comparing TF-IDF models trained on different corpora as if weights are comparable.
  • Ignoring the vocab size explosion from n-grams — 2-grams over 50k vocab can be tens of millions of columns.
Root cause analysis

Common issues

Error
Everything classified into one class (vocab mismatch)
Root cause

Train and serve vocab differ because vectorizer refit or vocab pruning skipped

Fix

Fit once, persist the vectorizer, and treat <UNK> with fixed terms at serve.

Error
Model too sparse to learn (huge vocab, few docs)
Root cause

No max_features/min_df pruning

Fix

Cap max_features (10–50k), set min_df=2, and enable sublinear_tf.

When things break

Debugging tips

  • Print the top-coefficient words per class; a junk token topping the list means filtering failed upstream.
  • Sparsity ensures a bug: a dense result with 0s everywhere suggests the model fed an all-UNK corpus.
  • If a class is predicted by default, check vocab coverage per class — Naïve Bayes needs identical vocab.
  • Inspect IDF values: a token with max ID (≈1 unique doc) is probably a typo or <UNK> remnant.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A term appearing in one document out of a corpus of 1,000 gets an IDF that is:

2

TF-IDF must fit on the full corpus including the test set for best performance.

Go deeper

Official documentation

sklearn TF-IDF vectorizer
docs
TF-IDF deep dive
docs