Classification
Intermediate 13 min read Classification

Naive Bayes

Counting wins when dimensions explode

Bayes' rule plus a naive conditional-independence assumption yields classifiers that are extremely fast and effective on high-dimensional, sparse, text-like features.

naive bayesbayes ruleconditional independenceMultinomialNBspamtext classification
Prerequisites: Classification Overview
Before we start

What you will learn

  • Derive NB from Bayes' rule and the naive assumption
  • Use MultinomialNB/GaussianNB on the matching feature types
  • Explain where NB surprises (correlated features, calibrated odds) and where it dominates
Introduction

What is this?

Naive Bayes applies Bayes' theorem: P(class|x) ∝ P(class) ∏_j P(x_j | class), assuming features are conditionally independent given the class. It estimates each feature-class table by counting, so training is O(lines × features) and it needs far fewer samples than parametric boundaries — the classic spam/text/NLP starter.

Motivation

Why do you need it?

  • Count-based estimates are fast, robust on sparse data and need no scaling.
  • It often matches or beats logistic on bag-of-words text for pennies of compute.
  • The math is the most relatable probability model: it explains what learning from counts is.
  • MultinomialNB is the blessed baseline for anything token-based.
Think about it

Mental model

Imagine

A doctor's back-of-envelope priors

Think of

Naive Bayes

'Most coughs are colds (prior); with fever AND rash, think measles (likelihoods).' The doctor multiplies like independent clues — exactly NB's counting, with its known habit of double-counting correlated clues.

Mechanics

How it works

  • Estimate priors P(class) and likelihoods P(x_j|class) from counts (with Laplace smoothing to avoid 0).
  • Score a row by multiplying prior x per-feature likelihoods (in log space!).
  • GaussianNB for continuous features (estimates per-class mean/std); MultinomialNB for counts (tokens); BernoulliNB for binary flags.
  • Classify by argmax class posterior.
  • The naive assumption is often false but the estimator still sorts classes well.

Bayes-rule classifier flow.

Prior
Likelihood
Multiply
Argmax
Hands-on

Examples

MultinomialNB on a toy token matrix

MultinomialNB on a toy token matrix
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
docs = [
    "free money now", "limited offer free", "claim your prize",
    "meeting agenda", "note on release", "thanks for attending",
]
y = np.array([1, 1, 1, 0, 0, 0])
 
X = CountVectorizer().fit_transform(docs)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=1)
 
m = MultinomialNB().fit(Xtr, ytr)
print("acc:", round(accuracy_score(yte, m.predict(Xte)), 2),
      "log-prob classes:", np.round(m.class_log_prior_, 2))

Counts of tokens drive posteriors directly; even on a six-doc toy set the sparse count matrix trains instantly and scores sensibly.

Real world

In production

  • Spam/phishing: MultinomialNB on token counts as the cheap first gate.
  • Support routing: keyword-count classifier triage before deeper models.
  • Sentiment quick-probe: BernoulliNB on positive/negative word flags for dashboards.
Guidelines

Best practices

  • Use MultinomialNB on integer counts (add-one/alpha smoothing); log-space scoring built in; for correlated tokens expect overconfident probabilities—calibrate if absolute probabilities matter.
Watch out

Common mistakes

  • Feeding raw float features to MultinomialNB (expect counts); forgetting alpha on sparse vocab (0s kill likelihoods); trusting NB's absolute probabilities (calibrate); expecting NB to win when features are highly correlated.
When things break

Debugging tips

  • NaN from log(0): raise alpha or check for empty vocab.
  • If accuracy plateaus, the naive independence is too strong — escalate to logistic/trees.
  • Check class_log_prior_ warnings when a class has zero training rows.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The 'naive' assumption is:

2

MultinomialNB expects features that are:

Go deeper

Official documentation

Naive Bayes (sklearn)
docs
Naive Bayes classifier | Wikipedia
docs
Naive Bayes tutorial (video)
video