Intermediate 12 min read Decision Trees

Decision Tree Overfitting

The loudest overfitter in the family

Trees overfit loudly: depth + collinear features + brittle single threshold paths. Cap, prune, forest-ify — but first read the sign (depth vs validation gap).

overfittingtree instabilitydepthvariancepruningnoise

Prerequisites: Tree Stopping and Pruning

Before we start

What you will learn

Identify over-growth from train/validation divergence
Explain why trees are high-variance learners
Apply caps, pruning and forest aggregation as fixes

Introduction

What is this?

A single tree is a high-variance learner: it re-traces its whole partition when one row wobbles, and un-capped growth memorizes training noise into deep singleton leaves. Symptoms: training accuracy near perfection, validation decaying as depth grows, fragile paths. Fixes: caps/leaf floors, cost-complexity pruning, and randomized forests (bagging averages the variance away).

Motivation

Why do you need it?

Recognizing overfit trees keeps you from misreading 'depth = power'.
The variance tale is the prelude to forests AND boosting — both exist to fix it.
Production trees that flip on one row are the #1 'the model changed overnight' complaint.
Leaf-noise fitting wastes decisions: a 3-row rule rarely survives contact with the future.

Think about it

Mental model

Imagine

Overfitting a puzzle by memorizing the corner-piece fits

Think of

Unconstrained tree growth

Deep leaves memorize individual rows the way cramming memorizes review questions: flawless on the seen set, clueless off it.

Mechanics

How it works

Grow to default; observe train accuracy in the high-90s with validation lagging — the sign.
Sweep depth/leaf constraints; watch validation move (the optimizer valley).
ccp_alpha prune: trim subtrees that only shrink train error.
Check stability: refit on 90% subsamples; path differences = variance.
Escalate to a forest when single-tree variance persists after pruning.

Tree error curves.

Shallowboth errors high (bias)

Sweet spotvalidation min

Deeptrain ~0, validation climbs

Prunepush back to the valley

Hands-on

Examples

Train vs test as the tree deepens

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
X, y = make_classification(n_samples=800, n_features=15, n_informative=8,
                           flip_y=0.12, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
 
for depth in [1, 3, 8, None]:
    m = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(Xtr, ytr)
    print(f"depth={depth}: train {accuracy_score(ytr, m.predict(Xtr)):.3f}"
          f"  test {accuracy_score(yte, m.predict(Xte)):.3f}")

The gap between train and test widens with depth — the algorithm is learning noise it will be graded on later.

Real world

In production

Release gates: exhaustive metrics + a depth/leaf audit before any tree-staged rollout.
Retain sanity: refit the same tree on the same data twice — identical params, same data => identical; any else is a seed/refactor bug.
Champion/challenger: compare pruned vs default trees live before promoting a config.

Guidelines

Best practices

Never release a bare-default tree; cap noise-fitting with leaf-size floors; quote validation numbers alongside train; treat 'train 100%' as a warning sign, not a badge.

Watch out

Common mistakes

Equating low depth with 'safe' (min_samples_leaf matters too); using test set to tune (leak); claiming stability without refit tests; forgetting trees may re-trace when categorical levels shuffle.

When things break

Debugging tips

Validation exact-100% on an easy problem => check for label leakage or duplicated rows across splits.
Path flips between two 90%-splits => the tree is variance-dominated; prune or forest.
Importance list re-orders every seed => structural instability, prune first.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

The classic overfit sign for a tree is:

Randomized averaging in forests fixes chiefly:

Go deeper

Official documentation

Instability of trees (sklearn guidelines)

docs

Bias-variance tradeoff | Wikipedia