Intermediate 13 min read Random Forest

Feature Importance and Stability

Read the compass, not the ledger

Forest importances mix split gains and permutation estimates — useful for ranking, biased by cardinality and collinearity. Read them as a compass, not a ledger, and validate with permutation or SHAP.

feature importancepermutation importancestabilitycollinearitySHAPranking

Prerequisites: OOB Score and Validation

Before we start

What you will learn

Distinguish gain-based and permutation importances
Explain cardinality and collinearity biases
Assess importance stability across seeds and builds

Introduction

What is this?

Two forest importance engines exist: MDI (mean decrease in impurity — cheap, built-in, but biased toward high-cardinality and collinear features) and permutation importance (randomize a column, measure score drop — honest but more expensive). Instability between retrains flags fragile rankings. For per-prediction magnitudes, apply exact SHAP tree explainer on the same forest.

Motivation

Why do you need it?

Ranking features is the #1 use of forests before feature engineering spends time.
Reading MDI as effect size actively misleads modeling teams downstream.
Stability between two seeds tells you whether the ranking is real at all.
SHAP on forests reconciles global lists with per-customer explanations.

Think about it

Mental model

Imagine

A vote-count leaderboard vs exit-poll impact estimates

Think of

MDI vs permutation/SHAP importance

Vote share (MDI) tells who gets airtime; exit-polls (perturbation) tell who actually changed outcomes. A celebrity with large airtime can move the least votes.

Mechanics

How it works

Gain importance: sum normalized impurity decrease each feature drove, per tree, averaged.
Permutation importance: shuffle one feature's values, measure OOB/validation score drop; repeat per feature.
Collinearity: correlated twins split credit arbitrarily in both — shuffle one and the other compensates.
Stability: refit with a different seed or bootstrap; compare rank correlation.
Bolt-on SHAP: shap.TreeExplainer over the forest for per-row additive attributions.

Ranking pipeline.

MDIfast, biased by cardinality

Permutationhonest perturbation per column

Stabilityrank check across seeds

Reportcombo chart in model card

Hands-on

Examples

MDI vs permutation on collinear features

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
 
rng = np.random.RandomState(0)
X, y = make_classification(n_samples=1200, n_features=8, n_informative=5,
                           random_state=0)
X = np.c_[X, X[:, 0] + rng.normal(0, 0.1, X.shape[0])]   # twin of feature 0
X = np.c_[X, X[:, 0] + rng.normal(0, 0.1, X.shape[0])]   # another twin
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
m = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
print("MDI (gain):", np.round(m.feature_importances_, 3))
perm = permutation_importance(m, Xte, yte, n_repeats=10, random_state=0)
print("Permutation:", np.round(perm.importances_mean, 3))

The twins split MDI credit three ways, each looking mid-tier — while permutation (and SHAP) reveals the cluster's true, undivided importance.

Real world

In production

Quarterly feature triage: rank with permutation, not MDI-alone, before dropping anything.
Model cards: publish both MDI and permutation so reviewers do not over-read the cheap one.
Per-customer support: TreeExplainer SHAP attributes each decision to real column values.

Guidelines

Best practices

Prefer permutation importance when ranking for cuts; report stability (rank-correlation across two seeds); distrust importance splitting among correlated features — group them; pin the same random_state across the ranking runs.

Watch out

Common mistakes

Commutating importance into causality; letting one id/high-cardinality column top the MDI chart silently; trusting a single sparse forest's ranking without OOB sanity; deleting 'low importance' features before validating that a train-after-deletion contract is stable.

When things break

Debugging tips

One giant MDI spike => inspect the column (ids, leakage, cardinality explode).
Permutation and MDI disagree on order => collinearity or large cardinality lurking.
Ranking flips by seed => the ranking is not signal; increase trees/samples first.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

Which importance prefers genuinely stable, causal-ish signal?

Two correlated twins each show low importance though the signal is real. Fix:

Go deeper

Official documentation

Permutation importances (sklearn)

docs

SHAP TreeExplainer