Random Forest
Advanced 14 min read Random Forest

Tuning Random Forests

Four knobs, done

Tune deliberately: set n_estimators to the OOB knee, sweep max_features, cap tree depth and leaf pop for latency, then verify with CV — a 4-knob guide to 95% of forest performance.

hyperparameter tuningn_estimatorsmax_featuresgrid searchrandomized searchcv
Prerequisites: Random Feature Subsets
Before we start

What you will learn

  • Order the tuning knobs by impact
  • Tune with OOB + final CV rather than randomized blast radius
  • Trade accuracy for latency on serving workloads
Introduction

What is this?

Random Forest is forgiving: four knobs carry almost all the weight. n_estimators buys variance reduction until its OOB knee; max_features trades per-tree strength for de-correlation; tree depth and min_samples_leaf contain growth and latency; min_impurity_decrease and class_weight patch weak tails. Grid/randomized search an ordered small grid; repeat a couple seeds; let CV bless the winner.

Motivation

Why do you need it?

  • Most teams over-tune: a small grid beats a full 100-point search on forests, at a tenth the compute.
  • OOB gives you a near-free scorer for the sweep loop.
  • Latency at serving: deep unpruned trees are read on many nodes per query; caps matter in p99 world.
  • Class imbalance: class_weight/sample_weight beat blind threshold-grinding.
Think about it

Mental model

Imagine

Tuning a radios you can hear vs a knob-bespoke instrument

Think of

Simple knobs-first tuning

On a forest, coarse knobs (trees, features, depth) do the heavy lifting; micro-details matter far less than the discipline of a held-out check.

Mechanics

How it works

  • Fix a seed; set n_estimators from the OOB-vs-tree-count knee.
  • Sweep max_features ∈ {1, sqrt(p), p/3, p} against OOB; keep the best neighborhood.
  • Sweep max_depth ∈ {None, 10, 6} and min_samples_leaf ∈ {1, 5, 20} — watch OOB and peak latency.
  • Re-run the top config twice with other seeds to confirm stability.
  • Bless via stratified CV on the actual pipeline (preprocessors included).

Knob order matters.

n_estimators
max_features
depth+leaf
CV
Hands-on

Examples

A tight, ordered search

A tight, ordered search
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.model_selection import GridSearchCV
 
X, y = make_classification(n_samples=2000, n_features=16, n_informative=8,
                           flip_y=0.1, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
 
grid = {
    "n_estimators": [100, 300],
    "max_features": [1, 4, 8],
    "max_depth": [None, 12],
    "min_samples_leaf": [1, 5],
}
gs = GridSearchCV(RandomForestClassifier(random_state=0, n_jobs=-1), grid,
                  cv=3, scoring="roc_auc", n_jobs=-1)
gs.fit(Xtr, ytr)
print("best:", gs.best_params_, "AUC", round(gs.best_score_, 4))
print("test AUC:", round(gs.score(Xte, yte), 4))

A small ordered grid over four knobs explores the space forests actually care about — OOB/CV inside, one holdout verdict outside.

Real world

In production

  • Automated retraining: schedule the same small grid weekly; compare OOB deltas before blue-ing the new champion.
  • Serving regression: cap depth to keep per-query node walks short; measure p99 before and after, not just accuracy.
  • Imbalanced fraud tails: sweep class_weight before over-sampling.
Guidelines

Best practices

  • Tune in dependency order; make early sweeps cheap (fewer trees), confirm the winner at full n_estimators; validate tuning on a fixed holdout only once; record seeds and grid in the model card.
Watch out

Common mistakes

  • Full-grid blast radius with tiny fold counts (overfit the search); tuning on the test set; doubling trees endlessly while forgetting the informative knobs; ignoring class_weight on imbalanced data then thresholding blindly.
When things break

Debugging tips

  • Best config at a grid edge => extend the range, don't settle.
  • All configs tie => the data is easy or the metric is coarse; switch the metric.
  • Winner doesn't reproduce across seeds => drop that config; pick the stable neighbor.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

n_estimators keeps helping until:

2

Tuning should ideally ping which scorer?

Go deeper

Official documentation

Ensemble parameters (sklearn)
docs
sklearn.model_selection help
docs
Practical random forest FAQ (Berkeley)
docs