Random Forest
Intermediate 11 min read Random Forest

Random Feature Subsets

The control knob for tree diversity

max_features tells the forest how many features each split may see — the lever that de-correlates trees, trades strength for diversity, and quietly drives regression vs classification defaults.

max_featuresfeature subsamplingsubsamplingdiversityde-correlationtuning
Prerequisites: Bagging and Bootstrap
Before we start

What you will learn

  • Explain the role of max_features in the forest
  • Set sane defaults for classification vs regression
  • Tune it deliberately rather than by habit
Introduction

What is this?

At every split, the forest considers only max_features randomly chosen features. Classification often uses sqrt(p); regression p/3. Small max_features => short-sighted trees, lots of diversity, low variance, but more bias per tree. Large => each tree is stronger but more correlated, so averaging gains less. The knob is the forest's central bias-variance trade.

Motivation

Why do you need it?

  • Randomness here is THE reason trees de-correlate; without it a forest collapses toward bagged single-decision trees.
  • The default differs by task — worth knowing why before you retune.
  • max_features is usually the single most influential forest hyperparameter.
  • It also caps compute: fewer candidates per node = cheaper splits.
Think about it

Mental model

Imagine

Hiring a committee where each member reads only a random few pages of the brief

Think of

Feature subsets

Everyone sharp on small slices; the combined verdict is diverse yet each member is individually fallible — averaging extracts the signal.

Mechanics

How it works

  • Each node samples max_features features at random (replacement irrelevant for weights).
  • Split scoring considers only those candidate columns.
  • Smaller max_features => weaker but more varied trees.
  • Classification default sqrt(p); regression p/3; 'sqrt', 'log2', None all available.
  • Combine with n_estimators: more diversity justifies more trees to average it.

Lowering max_features.

wide candidates
narrower
diversity
averaging
Hands-on

Examples

max_features sweep

max_features sweep
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_digits
 
d = load_digits()
for mf in [1, 2, 5, 16, 64]:
    s = cross_val_score(RandomForestClassifier(n_estimators=150, max_features=mf,
                        random_state=0), d.data, d.target, cv=5)
    print(f"max_features={mf:>2}  acc {s.mean():.4f}")

Small max_features loses because each tree is too weak; max_features = p approaches a plain bagged tree forest, losing the diversity advantage. The quadratic-ish peak is the sweet spot.

Real world

In production

  • Feature-heavy fraud sets often like lower max_features (narrow focus, honest trees).
  • Regression targets with many weak columns respond to slightly-larger-than-sqrt subsets.
  • Latency-sensitive serving: lower max_features cuts node cost with modest accuracy loss if needed.
Guidelines

Best practices

  • Start with the defaults (sqrt/log2 for classification, p/3 for regression); sweep max_features across a small grid alongside n_estimators; expect ties — don't over-rotate a third-of-a-percent tweak.
Watch out

Common mistakes

  • Forgetting regression defaults differ from classification; setting max_features=1 blindly; tuning max_features but leaving n_estimators tiny, so diversity never gets averaged; assuming max_features changes importance rankings (it does — check before quoting).
When things break

Debugging tips

  • Accuracy flat across the whole max_features sweep => the signal is in a few strong features; consider feature selection instead.
  • Forest very slow => max_features close to p on a huge p; shrink it.
  • Importance list reshuffles with max_features => rank with several sweeps before trusting it.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

max_features influences mostly which ensemble property?

2

Setting max_features too small risks:

Go deeper

Official documentation

RandomForestClassifier max_features doc
docs
Parameter choice (scikit-learn guidelines)
docs
Random forest theory (lecture notes)
docs