Decision Trees
Intermediate 13 min read Decision Trees

Tree Stopping and Pruning

Cap the variance, tune the complexity

Growth must stop somewhere: max_depth, min_samples_split, min_samples_leaf and impurity limits cap variance — and cost-complexity pruning trims a grown tree by a tuning knob ccp_alpha.

pruningmax_depthmin_samples_leafccp_alphacost-complexitystop criteria
Prerequisites: How Splits Work
Before we start

What you will learn

  • Set the standard stopping knobs and read their effects
  • Explain cost-complexity pruning and choose ccp_alpha via CV
  • Recognize the train/valid gap that signals over-growth
Introduction

What is this?

An unpruned tree keeps splitting until leaves are pure — memorization. Stopping criteria constrain growth: max_depth, min_samples_split (nodes below this won't split), min_samples_leaf (a leaf must hold >= this) and a purity/impurity improvement floor. Cost-complexity pruning instead grows the full tree then trims subtrees whose gain doesn't pay for their complexity, tuned by ccp_alpha.

Motivation

Why do you need it?

  • Depth caps are the tree's variance dial; without them validation dies.
  • min_samples_leaf is the sneaky-good knob for noisy tabular data.
  • ccp_alpha gives a principled trade between fit and size, CV-swept like any hyperparameter.
  • Audit decks need the same 'how deep, how constrained' numbers every review.
Think about it

Mental model

Imagine

A train that keeps stopping at every crossing vs the express

Think of

Over-grown vs pruned trees

Stopping at every crossing (pure leaves) moves slowly and memorizes the route; running express (pruned) reaches the same destinations with far fewer stops and generalizes to the next route.

Mechanics

How it works

  • Set constraints BEFORE fit: max_depth (e.g. 5-12), min_samples_leaf (e.g. 1-20), min_samples_split.
  • Fit; watch train vs validation accuracy — a booming train/valid gap is over-growth.
  • Cost-complexity: compute alpha path (alpha = impurity improvement minus node cost); prune subtrees where alpha drops below ccp_alpha.
  • CV-sweep ccp_alpha (and leaf sizes) to the validation sweet spot.
  • Record the chosen tree shape (depth, leaves, ccp_alpha) in the model card.

Stopping knobs at a glance.

max_depth
min_samples_leaf
impurity floor
ccp_alpha
Hands-on

Examples

Depth cap and ccp_alpha on validation

Depth cap and ccp_alpha on validation
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
X, y = make_regression(n_samples=1200, n_features=8, noise=25, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=1)
 
for depth in [None, 3, 6, 12]:
    m = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(Xtr, ytr)
    print(f"depth={depth}: train {mean_squared_error(ytr, m.predict(Xtr)):.0f}"
          f"  test {mean_squared_error(yte, m.predict(Xte)):.0f}")
 
m = DecisionTreeRegressor(ccp_alpha=8.0, random_state=0).fit(Xtr, ytr)
print("ccp_alpha=8: depth", m.get_depth(), "test MSE",
      round(mean_squared_error(yte, m.predict(Xte)), 0))

Depth and ccp_alpha both pull the same string: more constraint = worse train, usually better test once over-growth is tamed.

Real world

In production

  • Leaf-size floors prevent a dozen-row rules from driving production decisions.
  • ccp_alpha-swept trees give a single, defensible complexity number for audit decks.
  • Shallow pruned trees avoid 'the model changed because one row flipped' disasters.
Guidelines

Best practices

  • Always set SOME constraint (never bare defaults to full depth); tune depth plus leaf-size jointly; prefer ccp_alpha when you want a continuous complexity dial; cross-validate the chosen geometry.
Watch out

Common mistakes

  • Forgetting constraints entirely (default overfit); only raising min_samples_leaf without touching depth; pruning on the test set; reading 'depth 20' as a quality signal.
When things break

Debugging tips

  • Validation flat regardless of ccp_alpha => the data has little to prune; keep the simplest tree.
  • Huge leaf count with good validation => check inference latency at serving.
  • A pruned tree that trims half its leaves with no loss => the depth was always noise; simplify the deck.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

min_samples_leaf prevents:

2

ccp_alpha works by:

Go deeper

Official documentation

Tree pruning with ccp_alpha (sklearn)
video
Minimal cost-complexity pruning | Wikipedia
docs
DecisionTreeRegressor params (sklearn)
docs