Model Explainability
Intermediate 14 min read Model Explainability

Feature Importance

Which columns run the show?

Feature importance ranks how much each feature drives predictions: coefficients for linear models, impurity-based or permutation importance for trees, and drop-column tests for anything. Permutation importance is model-agnostic and honest about collinearity.

feature importancepermutationimpuritydrop-columnmodel-agnosticmean decrease accuracy
Prerequisites: Explainability Overview
Before we start

What you will learn

  • Compute permutation importance and interpret its stability
  • Explain why impurity importance is biased
  • Handle collinear features in importance rankings
Introduction

What is this?

Feature importance measures how much each feature contributes to the model's performance or predictions. Methods: linear coefficients (with scale caveats), tree impurity gains (fast, biased toward high-cardinality features), permutation importance (shuffle the feature, measure score drop — model-agnostic and honest), and drop-column tests (retrain without the feature — costly but causal-ish). Rankings answer 'what does the model rely on?', and anomalies are often bugs or leaks.

Motivation

Why do you need it?

  • Prioritizes: prune dead features, cut pipeline cost, focus feature work.
  • Debugging: a leaked/stale feature lights up as unlikely top importance.
  • Compliance: proves which inputs actually drive decisions.
  • Communication: stakeholders understand 'these three things matter' instantly.
Think about it

Mental model

Imagine

A chef asking which ingredient sells the dish

Think of

Feature importance

Swap the garlic for something neutral and watch sales — a big drop means garlic mattered (permutation). If you just count how often the recipe mentions garlic, you over-rate it (impurity bias). And 'remove the salt and re-cook the whole menu' tells you the true menu-level impact (drop-column).

Mechanics

How it works

  • Coefficients: standardize features first so β sizes are comparable; sign = direction.
  • Impurity-based (tree models): sum Gini/entropy gains across splits; biased to many-valued features.
  • Permutation importance: for each feature, shuffle its column, re-score, record drop from baseline.
  • Drop-column: retrain without the feature and compare — accurate, expensive (n retrains).
  • Report with variance: averaged over several shuffles/seeds, plus standard error.

Permutation importance: shuffle one column, measure the score drop.

Baseline
Shuffle col j
Re-score
Repeat
Rank
Real world

In production

  • Feature curation in recurring retrains: auto-drop features that stay at zero importance.
  • Audit pack: top-20 importances snapshot per model version.
  • Data quality: a suddenly-top or suddenly-zero feature gets an alert.
Guidelines

Best practices

  • Standardize numeric features before interpreting linear coefficients.
  • Use permutation (not impurity) for tree importance in production reporting.
  • Compute importances on the validation split, never the training split, to avoid overfit artifacts.
  • Report a flat importance profile next to any 'winner' — this is collinearity's fingerprint.
Watch out

Common mistakes

  • Comparing impurity importances across correlated features and calling the diff meaningful.
  • Interpreting importance as causality — it is association-conditional on the rest of the features.
  • Permuting a high-cardinality column and blaming the model when the shuffle destroys structure.
  • Trusting a single-seed permutation report.
Root cause analysis

Common issues

Error
Permutation importance much lower than impurity importance
Root cause

Shuffle destroys even weakly-informative columns while tree gains double-count

Fix

Prefer permutation for reporting; note impurity remains useful for pure feature selection.

Error
Importance flips between runs
Root cause

Single shuffle seed + collinear features

Fix

Average over 10+ seeds and report std; treat near-ties as tied.

Error
Feature with zero variance scores highest
Root cause

Leak/constant edge case or categorical encoding artifact

Fix

Investigate the feature; constant features can't drive real model behavior.

When things break

Debugging tips

  • A top-1 feature that is an id/date/timestamp = probable leakage; audit its pipeline.
  • Two collinear features sharing ~50% importance each is expected — report the group, not the pair.
  • If permutation importance NEGATIVE for several features, your baseline may be mis-scored — fix the metric first.
  • Recompute after feature encoding changes; importance on raw vs one-hot features isn't comparable.
Interviews

Interview checkpoint

Practice

Hands-on lab

Rank Features Honestly
30 min · intermediate
In progress
Objectives
  • Compute linear-format importance on a toy dataset
  • Run permutation importance with sklearn
  • Compare it to tree impurity importance and reason about the gap

Rank Features Honestly

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Permutation importance is computed by:

2

Tree impurity-based importance is unbiased across feature cardinalities.

Go deeper

Official documentation

scikit-learn permutation importance
docs