Model Explainability
Advanced 17 min read Model Explainability

SHAP Values

Fairly dividing each prediction's credit

SHAP decomposes each prediction into additive per-feature Shapley values — how much each feature pushed the prediction away from the baseline. Consistent, locally accurate, and the default for explaining tabular models.

SHAPShapleyadditive attributionbaselinewaterfallfeature contribution
Prerequisites: Feature Importance Model Evaluation
Before we start

What you will learn

  • Explain Shapley values and additive decomposition
  • Read SHAP summary and waterfall plots
  • Recognize approximation caveats at scale
Introduction

What is this?

SHAP explains a prediction as a baseline (average prediction) plus per-feature additive contributions: predicted = base + Σ φⱼ, where φⱼ is the Shapley value of feature j — the fair average of its marginal contribution over all feature subsets/orderings. It is the only additive attribution with consistency guarantee, so it agrees with humans' intuition of 'pushed up vs down'. Computation is exact for trees and the kernel SHAP approximation for general models.

Motivation

Why do you need it?

  • Consistency: a model that relies more on a feature can't give it a smaller φ — rare among explainers.
  • Locally accurate: contributions sum exactly to the prediction vs baseline.
  • One framework for texts, images and tables via different backends.
  • The de-facto standard for tabular ML auditing and reporting.
Think about it

Mental model

Imagine

Handing out a bonus fairly across a team

Think of

Shapley value attribution

The bonus per teammate measures how much each contributed across every possible grouping — alone, in pairs, with the whole team. SHAP does the same for features: how much each pushed the prediction, averaged over all coalitions, guaranteeing the parts sum to the whole.

Mechanics

How it works

  • Define baseline φ₀ = E[f(x)] over the background dataset.
  • For feature j, compare f on subsets that include vs exclude j, averaged over all subsets → Shapley φⱼ.
  • Enforce efficiency: Σ φⱼ = f(x) − baseline — the waterfall closes.
  • Fast paths: TreeSHAP (exact for trees), linear SHAP (analytic), gradient SHAP for NN, KernelSHAP general.
  • Aggregate: |φ| averages → per-feature global importance; spatial/cross rows → dependence plots.

A SHAP waterfall: base + pushes = prediction.

Baseline
Feature A
Feature B
...
f(x)
Real world

In production

  • Adverse-action letters: top-3 negative contributions per rejected application.
  • Model audits: SHAP summary + dependence plots shipped with each model version.
  • Feature-aware monitoring: track average |φ| per feature over time — drift early warnings.
Guidelines

Best practices

  • Use TreeSHAP for tabular tree models; default is exact and fast.
  • Choose a background sample that reflects production distribution (e.g. recent 1000 rows).
  • Plot summary + dependence + waterfall together; one metric alone over-trusts a single view.
  • For correlated features, read grouped SHAP (sum over the correlated block).
Watch out

Common mistakes

  • Interpreting φ as causality or as what would happen if you changed the feature.
  • Running KernelSHAP on 10k-feature text and calling the noisy output meaningful.
  • Editing one feature while holding others fixed and expecting φ to track — SHAP is counterfactual-consistent but not simplistic.
  • Forgetting that baseline choice changes every value; report it.
Root cause analysis

Common issues

Error
Waterfall doesn't close (Σ φ ≠ f(x) − base)
Root cause

Approximated backend (KernelSHAP) under-sampled or baseline misspecified

Fix

Increase samples, verify background matches production, and use TreeSHAP when the model allows.

Error
SHAP values look random run to run
Root cause

Too few background samples for KernelSHAP

Fix

Grow background size, fix seed, or switch backends with a deterministic path.

Error
Negative importance for obviously useful feature
Root cause

Collinearity redistributes credit; lone-feature SHAP hides the group

Fix

Report grouped/correlated feature contributions and pair with permutation importance.

When things break

Debugging tips

  • Verify efficiency numerically: f(x) − base ≈ Σ φ on a handful of rows — if not, something's wrong in plumbing.
  • Two explainers (TreeSHAP vs PermutationSHAP vs KernelSHAP) should broadly agree; big gaps = approximation problems.
  • SHAP value near zero for a leak-y feature usually means the leak is redundant, not absent.
  • Plot |φ| vs feature value as a dependence plot — linear-ish bands suggest additive relationships.
Interviews

Interview checkpoint

Practice

Hands-on lab

Open a Model with SHAP
40 min · advanced
In progress
Objectives
  • Train a gradient-boosted tree and compute TreeSHAP values
  • Plot summary and a waterfall for one prediction
  • Verify the sums close and interpret the attribution

Open a Model with SHAP

40 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The sum of a sample's SHAP values equals:

2

SHAP's consistency guarantee means a model relying more on a feature must give it a non-smaller value.

Go deeper

Official documentation

SHAP paper (NIPS)
docs
SHAP docs (Slundberg)
docs