ML Fundamentals
Intermediate 12 min read ML Fundamentals

Overfitting and Underfitting

Memorize nothing, learn the pattern

Underfits miss the pattern, overfits memorize it — detect each by the train/validation gap and fix with capacity, data, regularization, and ensembles.

overfittingunderfittingcapacityregularizationearly stoppingtrain-validation gap
Prerequisites: Bias-Variance Tradeoff
Before we start

What you will learn

  • Diagnose under- vs overfitting from train/validation error trajectories
  • Apply capacity, data, regularization and early stopping fixes for each side
  • Explain the double-descent observation and its practical lesson
Introduction

What is this?

Underfitting: the model is structurally too weak to capture the relationship — high error everywhere. Overfitting: the model captures noise as if it were signal — training error collapses while validation error stagnates or rises. Practical control knobs: capacity (more/less complexity), data volume, regularization strength, early stopping, feature pruning and ensembling.

Motivation

Why do you need it?

  • Both failure modes are cheap to diagnose with a train-vs-validation error gap.
  • Correctly classified failures dictate different fixes — misdiagnosis wastes whole sprints.
  • Every real dataset has signal and noise; the model must respect the boundary between them.
  • A deployed overfit model is a time bomb: it decays fast as the world shifts.
Think about it

Mental model

Imagine

Cramming for a test by memorizing answers vs studying the subject

Think of

Overfitting vs learning the pattern

Cramming looks perfect on known questions and fails the real-world version of the exam — exactly how an overfit model behaves on new data.

Mechanics

How it works

  • Underfit signature: train error high AND validation error high — capacity too small, features weak, or model family wrong.
  • Overfit signature: train low, validation higher — capacity too large, variance-dominated.
  • Fix underfit: add capacity/features/deeper depth, relax regularization, revisit the feature set.
  • Fix overfit: more data, stronger regularization, smaller capacity, dropout/bagging, early stopping.
  • Watch the gap as you change one knob at a time; the gap is your compass.

Train vs validation error curves.

Early
Middle
Late
Stop
Hands-on

Examples

Depth sweep on a small dataset

Depth sweep on a small dataset
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
rng = np.random.RandomState(0)
X = np.linspace(0, 10, 150)
y = np.sin(X) + rng.normal(0, 0.35, 150)
Xtr, Xte, ytr, yte = train_test_split(X.reshape(-1, 1), y, test_size=0.4, random_state=1)
 
for depth in [1, 3, 6, 12]:
    m = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(Xtr, ytr)
    print(f"depth {depth:>2}: train {mean_squared_error(ytr, m.predict(Xtr)):.3f}"
          f"  test {mean_squared_error(yte, m.predict(Xte)):.3f}")

Depth 1 underfits (both high), deep trees overfit (train ~0, test high); depth 3-6 find the balance — classic capacity knob in action.

Real world

In production

  • Finance models are audited: report the train/valid gap so reviewers can sanity-check fit quality.
  • Retraining loops use early stopping on a validation slice to keep models general.
  • Feature teams who add columns without validation checks routinely ship overfit models — the gap audit catches them.
Guidelines

Best practices

  • Hold off on adding nontrivial model capacity until the data-to-noise ratio is understood; track the gap after every change; prefer ensembles and regularization over endlessly increasing depth; log gap metrics to the model card.
Watch out

Common mistakes

  • Reading only train accuracy; turning off regularization because train error rises; adding data to fix a bias (underfit) problem; ignoring class-imbalance effects that look like overfitting.
When things break

Debugging tips

  • Re-run with more data: if validation improves a lot, variance was the problem; if nothing moves, bias is.
  • Check if the model flips predictions on tiny input jitter — high variance models flap.
  • Inspect the worst validation errors: are they specific rows that also poison other models (data bugs, not overfit)?
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Underfitting signature:

2

Best first fix for an overfit tree?

Go deeper

Official documentation

Underfitting vs overfitting — sklearn example
video
Overfitting | Wikipedia
docs
Google: generalization and overfitting
video