ML Experimentation
Advanced 16 min read ML Experimentation

Statistical Comparison of Models

Is that 0.005 a win or weather?

Comparisons should be statements about signal, not luck: paired tests, bootstrap confidence intervals, McNemar for classifiers, and effect-size-aware thresholds replace '0.915 beats 0.910' with 'the gain exceeds the noise band'.

paired testconfidence intervalbootstrapMcNemareffect sizesignificance
Prerequisites: Sound Experiment Design Evaluation Metrics
Before we start

What you will learn

  • Build bootstrap CIs and paired differences
  • Run a paired test (e.g. McNemar) correctly
  • Report effect size and noise, not just the point estimate
Introduction

What is this?

Model comparison should quantify uncertainty around the DIFFERENCE. Approaches: paired resampling (bootstrap the test set by sample → CI on the metric delta), paired statistical tests (McNemar for misclassification counts on the same samples; paired t/Wilcoxon over CV fold metrics), and effect-size statements ('+0.004 AUC, 95% CI [−0.001, +0.009]'). A claim without a CI or test result is a point estimate, and point estimates lie.

Motivation

Why do you need it?

  • Small metric deltas are indistinguishable from noise; reporting them as wins misroutes roadmaps.
  • Paired designs are dramatically more powerful: the same data points in both groups cancel shared variance.
  • Compliance and leadership demand defensible evidence, not vibes.
  • It stops the treadmill: you stop chasing every 0.002 and only act on real gains.
Think about it

Mental model

Imagine

Two chefs judged by the same tasters, twice

Think of

Paired comparison

Both dishes are tasted by the SAME panel on both days. That pairing removes 'the panel was tired today' noise, so a consistent 2-of-10 preference shrug becomes a real signal. Different panels (unpaired) would swallow the difference in taster mood.

Mechanics

How it works

  • Generate both models' predictions on the SAME test samples (pairing).
  • Bootstrap: resample test indices ~1000×; each resample → both models' metric → a delta distribution.
  • Report the delta CI: if it excludes 0, the models differ; the width is your certainty.
  • For classification, McNemar's test on the discordant cells (A right ∧ B wrong vs A wrong ∧ B right).
  • For repeated CV, paired t-test/Wilcoxon over the K fold scores of A vs B (same folds).

Bootstrap the delta; check the CI against zero.

Same samples
Resample
Metric A−B
CI of Δ
Verdict
Real world

In production

  • Model choice gates: adopt only when the lift CI clears a pre-agreed threshold.
  • A/B on the backend: paired serving cohorts with the same users across buckets.
  • Release review: every model change ships a bootstrap CI for its headline metric.
Guidelines

Best practices

  • Always compare on the SAME indices — unpaired comparisons double the noise.
  • Pre-commit thresholds (e.g. lower 95% CI bound must exceed +0.003) before running.
  • Report effect + CI + N together; a CI without N is unreadable.
  • Use McNemar for classifiers, paired fold tests for CV curves, and bootstrap for metric deltas.
Watch out

Common mistakes

  • Reporting test-set point scores without error bars.
  • Using unpaired t-tests or independent bootstrap when the data are naturally paired.
  • Multiple-testing orgies across 30 metrics without correction → a false positive win.
  • Concluding equivalence from a non-significant p — absence of evidence ≠ evidence of absence.
Root cause analysis

Common issues

Error
CI absurdly wide (spanning ±0.05)
Root cause

Small test N or high-variance metric

Fix

Grow the test set or use variance-reduction tricks: stratify resampling, paired deltas, and larger samples.

Error
McNemar refuses to run (assumption violated)
Root cause

Discordant cells too small or tied predictions dominate

Fix

Fall back to the bootstrap CI on the delta; note that tiny discordance means the difference is negligible.

Error
Significant on 1 of 20 metrics, none others
Root cause

Multiple testing

Fix

Pre-register the primary metric; apply Bonferroni/Holm if you must sweep.

When things break

Debugging tips

  • Shuffle-the-labels sanity: the delta distribution should center near zero; if not, your paired logic leaks.
  • Check CI width vs delta: width > delta means you need more power (data/samples), not more reports.
  • Run the test on a no-difference pair (A vs A) — it should say 'tie' every time.
  • Log the resample indexes' seed so every CI is replayable.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A bootstrap CI for a paired metric delta is built by:

2

A non-significant difference proves the models perform equally.

Go deeper

Official documentation

Comparing models (scikit-learn)
docs