Model Explainability
Intermediate 15 min read Model Explainability

Explainability Overview

Opening black boxes without opening the weights

Explainability answers why a model makes its decisions: global (what the model learned overall) and local (why this prediction). It covers feature importance, SHAP, LIME, surrogates, PDPs and counterfactuals — each with different guarantees and costs.

explainabilityinterpretabilityglobal vs localfeature importanceSHAPLIMEsurrogate
Prerequisites: Model Evaluation Feature Engineering
Before we start

What you will learn

  • Distinguish global vs local explanations
  • Contrast inherently interpretable vs post-hoc models
  • Match a question (why trust? why reject? which feature?) to a technique
Introduction

What is this?

Explainability refers to methods that describe what a model does. Global explanations summarize model behavior over the whole dataset (feature importance, PDPs, surrogate trees); local explanations explain a single prediction (SHAP values, LIME, counterfactuals). Some models are inherently interpretable (linear, trees, rules); black boxes get post-hoc explanations. The right tool depends on the question: audit a feature, justify a decision, debug a drift, or serve a regulation.

Motivation

Why do you need it?

  • Regulatory and risk contexts (credit, health, insurance) demand decisions you can justify.
  • Debugging: explaining a bad cluster of predictions reveals data leaks and encoding bugs.
  • Trust: stakeholders (and users) act on models they can verify.
  • Feature attribution quantifies what the model actually relies on — catching shortcut learning.
Think about it

Mental model

Imagine

A doctor explaining your diagnosis

Think of

A model explaining its prediction

You don't need the doctor to hand over their textbook chapters; you want to know which symptoms and tests drove the verdict. Local explanation = 'these three findings drove this diagnosis'. Global explanation = 'across 10,000 patients, fever and rash are the strongest drivers'.

Mechanics

How it works

  • Inherently interpretable: linear coefficients, tree splits, rule lists — readable by construction.
  • Global post-hoc: permute each feature and measure the drop in performance → importance.
  • Local attribution: SHAP decomposes a prediction into per-sample additive contributions.
  • Local surrogate: LIME fits a small interpretable model around a single prediction.
  • What-if & counterfactual: perturb features and ask 'how much change to flip the outcome?'.

Explainability questions and their tools.

Why this prediction?
What drives the model?
How does feature X behave?
What would flip it?
Audit + report
Real world

In production

  • Credit scoring: SHAP waterfall for adverse-action letters and audit trails.
  • Medical triage: explaining model flags before clinician review.
  • Churn systems: segmenting WHO is leaving from WHY via global importance.
Guidelines

Best practices

  • Define WHO needs the explanation (auditor, user, engineer) before picking a tool.
  • Pair global importance with local explanations — each alone over-simplifies.
  • Sanity-check explanations against domain knowledge; surprising attributions surface bugs.
  • Report uncertainty: SHAP approximations and LIME samples have variance.
Watch out

Common mistakes

  • Reading correlation-heavy importance (colinear features) as causal.
  • Using a local explanter to describe the whole model.
  • Trusting black-box attributions without cross-checking with a second method.
  • Explaining a proxy (training metric) instead of the production target.
Root cause analysis

Common issues

Error
SHAP and permutation importance disagree
Root cause

Collinear features + different attribution semantics

Fix

Note that SHAP splits credit fairly across correlated features while permutation drops them jointly; report both with a collinearity warning.

Error
Explanations unstable across samples
Root cause

Local surrogate/approx sampling noise or high variance

Fix

Grow LIME sample counts, fix seeds, aggregate repeated explanations before trusting a single one.

Error
Importance dominated by one useless feature
Root cause

Feature leak (id leak or target leakage)

Fix

Investigate the feature pipeline; check the feature's profile vs label.

When things break

Debugging tips

  • Run two independent explainers (e.g. SHAP + permutation) — disagreement flags fragile attributions.
  • Explain a handful of known-good predictions first; the tools should confirm your intuition.
  • Check leakage: high importance for a personal-ID or date feature often means a leak.
  • Stress with duplicates: feature importance that spikes on near-duplicate rows signals collinearity.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Explaining a single rejected loan application is:

2

Linear regression coefficients are inherently interpretable.

Go deeper

Official documentation

Interpretable ML book
docs
SHAP library
docs