Model Explainability
Intermediate 15 min read Model Explainability

Surrogate Models & LIME

Fitting a tiny explainable model around one prediction

LIME builds a small interpretable model (sparse linear) around a single prediction by perturbing inputs and fitting a local region. Nice for images/text; risky when the local region isn't actually local.

LIMEsurrogatelocal surrogateperturbationkernel widthsparse linear
Prerequisites: SHAP Values
Before we start

What you will learn

  • Explain how LIME perturbs inputs and fits a local surrogate
  • Contrast LIME with SHAP on meaning and cost
  • Use LIME sensibly on tabular, text and image inputs
Introduction

What is this?

LIME (Local Interpretable Model-agnostic Explanations) explains a single prediction by: (1) generate neighbors by perturbing the input (tabular: jitter real features; image/text: toggle superpixels/word presence), (2) get the black-box's predictions for those neighbors, (3) fit a sparse interpretable model (linear/decision tree) weighted by locality from a kernel, and (4) read the surrogate's coefficients as the explanation. It's model-agnostic and popular for images/text; its Achilles heel is that the fitted linear region can be far from locally faithful.

Motivation

Why do you need it?

  • Works on ANY model — including language models and CNNs — without internals.
  • The surrogate is human-readable by construction: a small set of weighted features.
  • Fine-grained: can explain at word or super-pixel granularity.
  • A strong complement to SHAP (exact tabular attribution + readable local views).
Think about it

Mental model

Imagine

Surveying neighbors to describe one house's market appeal

Think of

Fitting a LIME surrogate

You can't understand the full appraisal model, but asking how similar-but-changed houses sell ('no pool', 'extra room', 'garage') lets you fit a little local price rule for this neighborhood — and the rule explains why THIS house is worth what it is. How well your 'local' neighborhood matches the real one decides whether the explanation holds.

Mechanics

How it works

  • Pick instance x; perturb features to build neighbors {x′ᵢ} (tabular: noise / sampling; image: mask super-pixels; text: hide words).
  • Predict neighbors with the black box → target values {y′ᵢ}.
  • Weight each neighbor by proximity kernel (exponential in distance; bandwidth is a knob).
  • Fit weighted sparse linear (or tree) surrogate on {x′ᵢ} → {y′ᵢ}.
  • Explanation = surrogate's top weighted features/coefficients for x.

Perturb → predict → fit local surrogate.

x
Neighbors
Black-box
Weighted fit
Coefficients
Real world

In production

  • Ad-hoc debugging of single model decisions and misclassification clusters.
  • Explaining image classifiers with semantic super-pixels for error analysis.
  • Document/favorite dashboards where ML teams inspect random samples.
Guidelines

Best practices

  • Check the surrogate's fit (R², agreement) on its neighborhood before trusting coefficients.
  • Set the kernel bandwidth from domain sense of 'locality'; too wide fits a global line.
  • For images, choose super-pixels large enough to be meaningful; tiny ones explode the search.
  • Use LIME for exploration/debugging; prefer SHAP for audited tabular reporting.
Watch out

Common mistakes

  • Treating surrogate weights as global feature importance.
  • Perturbing at a scale neighbors stop resembling real data — the 'local' fit is then global.
  • Fitting linear surrogate on a heavily nonlinear decision boundary (non-local region).
  • Ignoring surrogate fit quality and reading coefficients anyway.
Root cause analysis

Common issues

Error
Explanation flips between runs
Root cause

Perturbation sampling randomness + over-wide kernel

Fix

Fix the random state, raise neighbor count, and tighten the kernel bandwidth.

Error
Surrogate contradicts domain sense
Root cause

Neighborhood includes unrealistic feature combos (out-of-support)

Fix

Constrain perturbations to the data manifold (sample from training distribution, don't jitter blindly).

Error
LIME slow on many images
Root cause

Super-pixel segmentation + thousands of fits per image

Fix

Batch perturbations, cut neighbor counts, offset heavy explainers to offline analysis.

When things break

Debugging tips

  • Print the surrogate R²; if below ~0.6 the linear view is fabrication, not explanation.
  • Vary the bandwidth and watch whether the explanation flips — instability means the region isn't locally linear.
  • Explain 20 neighbors of the same class; consistent top features are trustworthy, inconsistent are noise.
  • Compare LIME vs SHAP on the same rows; gross disagreement signals a locality breakdown.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

LIME's explanation is derived from:

2

LIME works with models whose internals are completely hidden.

Go deeper

Official documentation

LIME paper
docs
lime library
docs