ML Experimentation
Intermediate 15 min read ML Experimentation

Error Analysis

A score is a summary; slice it before it stalls you

Error analysis digs into WHERE a model fails: which classes, segments or slices concentrate the mistakes? Confusion matrices, per-slice metrics and clustering failures turn a bad score into a prioritized punch list.

error analysisconfusion matrixclass imbalanceslice metricshard examplesfailure modes
Prerequisites: Evaluation Metrics Experiment Tracking
Before we start

What you will learn

  • Build confusion + per-class and per-slice error summaries
  • Cluster or sample failures to find common failure modes
  • Turn error analysis into ranked model/data actions
Introduction

What is this?

Error analysis breaks aggregate metrics into actionable slices: per-class precision/recall, per-segment rates (category, region, recency, feature ranges), and clusters of misclassified examples. It separates 'model is 0.90 AUC' from 'the model is blind to rare classes, to new regions, or to content-type X'. The output is a ranked list of fix candidates — data, features, or model — selected by impact and effort.

Motivation

Why do you need it?

  • Aggregate accuracy hides silent, high-impact failures on important subgroups.
  • Prioritizes work: fix the slice that matters most, first.
  • Constant, current practice: error analysis on every retrain catches data drift early.
  • Feeds everything else: annotation gaps, feature engineering ideas, monitoring thresholds.
Think about it

Mental model

Imagine

A post-mortem for every missed shot in a match

Think of

Error analysis

The team scored 70% — fine. But are the missed shots all in the last 5 minutes, all corner kicks, all against the same opponent style? That's the difference between 'practice more' and a targeted drill. Same ball game, opponent and match time decide the fix.

Mechanics

How it works

  • Collect predictions + ground truth + features/metadata on the held-out or production sample.
  • Confusion matrix → per-class precision/recall and dominant confusion pairs.
  • Slice by metadata/feature groups → per-slice error rates (deltas vs overall).
  • Embed + cluster or stratified-sample the errors → recurring failure modes.
  • Rank fixes by error-count × business impact and validate each on a small test.

From aggregate score to a prioritized fix list.

Predictions
Confusion matrix
Slice rates
Cluster errors
Ranked actions
Real world

In production

  • Class-imbalance triage: recovers the rare-class recall a global F1 hides.
  • Launch reviews: slice-level leaderboard answers 'does this work for our main segments?'.
  • Drift watchdog: slice accuracy vs the 30-day window alerts before aggregate scores blink.
Guidelines

Best practices

  • Define slices BEFORE looking at results (business segments, not whichever slice looks lucky).
  • Sample errors with metadata attached; labels-only samples are unreadable.
  • Quantify slice sizes: a big error rate on 5 examples is noise, not insight.
  • Track slice metrics over time; regressions per slice are your monitoring surface.
Watch out

Common mistakes

  • Reading overall accuracy and skipping the confusion matrix — hiding class failure.
  • Fixing a slice by adding labels without confirming the error is annotation-bound.
  • Reporting slice error rates without confidence/error bars at small N.
  • Slice-dredging until an interesting (but fake) pattern appears.
Root cause analysis

Common issues

Error
Overall good, slice terrible
Root cause

Imbalanced classes or underrepresented segment in training

Fix

Stratify sampling, upsample the slice, or add slice-balanced loss weighting.

Error
Big confusion between two classes
Root cause

Features discriminate poorly on their boundary

Fix

Add discriminative features or label-audit the boundary classes for mislabels.

Error
Error slices look random
Root cause

Slice definitions post-hoc/too granular

Fix

Pre-register slices and use wider bins; cluster errors for modes instead of slicing randomly.

When things break

Debugging tips

  • The dominant confusion pair usually names the fix: similar classes = disentangling features needed.
  • Errors concentrated in one date-range often signal data drift or a label pipeline change.
  • Clustering error embeddings repeatedly — stable clusters = real modes; drifting clusters = noise.
  • Slice errors that persist across seeds are real; single-seed artifacts aren't.
Interviews

Interview checkpoint

Practice

Hands-on lab

Slaughter Your Errors
35 min · intermediate
In progress
Objectives
  • Compute confusion matrix + per-class metrics
  • Slice error rate by a metadata column
  • Rank the output into a fix list

Slaughter Your Errors

35 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A confusion matrix's dominant off-diagonal cell tells you:

2

Slice error rates are trustworthy even with as few as 3 examples in the slice.

Go deeper

Official documentation

sklearn confusion_matrix
docs