Intermediate 12 min read Random Forest

Bagging and the Bootstrap

Resample, retrain, average

Bagging (bootstrap aggregation) retrains the model on resampled rows and averages — the statistical engine that turns a wobbling tree into a steady ensemble.

baggingbootstrapresamplingvariance reductionoobaggregation

Prerequisites: Random Forest Overview

Before we start

What you will learn

Explain bootstrap resampling with replacement
See why ~63% of rows appear per bootstrap sample
Trace bagging's variance-reduction argument

Introduction

What is this?

Bootstrap: draw a new dataset by sampling rows WITH replacement until you have n. Bagging: fit the base learner on each bootstrap sample and aggregate. A single deep tree is high-variance; bootstrap samples differ slightly, so their trees differ, and averaging shrinks variance. About (1 - 1/e) ≈ 63% of rows land in each sample; the rest are out-of-bag and act as free validation.

Motivation

Why do you need it?

It's the whole variance-reduction argument behind forests — say it in one breath.
OOB rows make honest validation redundant only after understanding what they are.
Averages of noisy-but-unbiased learners beat a single sharp-shooter in the long run.
Bagging generalizes: it de-noises ANY high-variance learner, not just trees.

Think about it

Mental model

Imagine

A mark hitting gold most shots, wobbling often vs taking the average of 100 shots

Think of

Single tree vs bagged average

The wobble (variance) averages out toward the gold; the average of 100 wobbling marks is a tight cluster.

Mechanics

How it works

Draw a bootstrap sample: n rows sampled with replacement from the train set.
Fit one deep tree on that sample.
Repeat for B trees (B = n_estimators).
Aggregate: majority class (or mean) over trees.
Rows never sampled feed OOB error = per-tree holdout aggregated like a real CV.

Bootstrap inclusion rate.

Drawn rows WITH replacement

Included~63% land once+

OOB~37% unseen = validator monkey

Averagetrees -> forest

Hands-on

Examples

OOB curve vs sample size

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
 
X, y = make_classification(n_samples=900, n_features=12, n_informative=6,
                           flip_y=0.1, random_state=0)
for n in [1, 5, 20, 100, 300]:
    m = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=0).fit(X, y)
    print(f"trees={n:>3}  OOB acc {m.oob_score_:.4f}  depth(avg) {int(np.mean(m.estimators_, dtype=object).get_depth()) if False else [t.get_depth() for t in m.estimators_[:1]]}")

OOB error stabilizes as tree count grows — the telltale flat-curve after which extra trees buy little. (The estimators_ depth snippet shows the deep, unpruned trees bagging relies on.)

Real world

In production

OOB as the early-warning accuracy monitor before you open a formal holdout.
Bagged regressors for noisy tabular targets where variance was the visible problem.
Change-simplified deploys: same params, new seed reproducibility so retrains diff cleanly.

Guidelines

Best practices

Monitor the tree-count-vs-OOB curve and stop doubling at the knee; keep bootstrap=True for variance reduction, only set False by conscious choice; log the OOB value next to the test score in the model card.

Watch out

Common mistakes

Believing rows not drawn are 'unused' (they ARE the OOB validator); setting n_estimators by round-number habit instead of the curve; tuning bootstrap=False off instinct; reading OOB as if it were a fresh independent holdout on small data (it shares all rows).

When things break

Debugging tips

Flat OOB curve from 50 to 500 => stop early; your trees are already steady.
OOB dramatically better than test => the OOB set is not independent of training signal; still trust the test as the final kidney.
If deep unpruned trees show bad OOB, then variance is NOT the only problem: check for label noise topology.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

A bootstrap resample contains roughly what fraction of unique rows?

Bagging mainly attacks which error component?

Go deeper

Official documentation

sklearn.ensemble.BaggingClassifier

docs

Bootstrap aggregating | Wikipedia