ML Experimentation
Intermediate 16 min read ML Experimentation

Sound Experiment Design

Engineering experiments you can actually believe

Experiment design keeps conclusions honest: one change at a time, fixed splits, baseline-first, multiple seeds, and the discipline to pre-declare the comparison before peeking at results.

experiment designbaselinesingle-variable changeseedpre-registeredcontrolstatistical
Prerequisites: Experiment Tracking Cross-Validation
Before we start

What you will learn

  • Structure comparisons: baseline, single variable, fixed splits
  • Use multi-seed runs to separate signal from noise
  • Pre-register the decision rule before the experiment
Introduction

What is this?

Sound experiment design makes results interpretable: (1) fix the data splits and evaluation protocol BEFORE experiments, (2) change one treatment at a time against a fixed baseline, (3) repeat across seeds/downsamplings so noise is quantified, and (4) pre-declare what 'better' means. This refuses the two deep-seated biases: comparing runs across different splits, and cherry-picking the run that happened to look best.

Motivation

Why do you need it?

  • Changing data + model + seed at once makes attribution impossible.
  • A baseline run is the reference frame — without it nothing is 'better'.
  • Multi-seed averages defeat the lottery-ticket illusion of lucky initializations.
  • Pre-declared decision rules keep you honest under result pressure.
Think about it

Mental model

Imagine

A clinical trial with a control arm and a protocol

Think of

ML experiment design

You never change the pill and the placebo and the intake criteria in the same trial. Same control group, one intervention, registered endpoints — and a conclusion you can defend. ML is the same: control = baseline model, intervention = one change, endpoints = pre-declared metrics.

Mechanics

How it works

  • Define fixed splits: train/val/test + cross-validation folds, hashed and frozen before any tuning.
  • Establish the baseline run first — best simple model with default/dev config.
  • Run one treatment per experiment: change a single hyperparameter, feature set, or preprocessing step.
  • Repeat each (baseline + treatment) over K seeds; report mean ± std with paired comparisons.
  • Apply the pre-declared rule: 'adopt only if mean val-AUC ≥ baseline + 0.005 via paired sign test'.

One variable, controlled, repeated across seeds.

Frozen splits
Baseline
Treatment
xK seeds
Pre-declared rule
Real world

In production

  • Feature-pruning decisions: does dropping column X really help?
  • Library/model upgrades: same pipeline, new estimator — provable gain or not.
  • Serve A/B lands on the same discipline: cured data, single treatment, paired metric.
Guidelines

Best practices

  • Freeze and hash the dataset version; any split drift invalidates every subsequent comparison.
  • Use paired comparisons (same folds/seeds) — they drown variance far faster than larger N.
  • Report mean ± std and p-value where regimes demand it; means alone hide noise.
  • Write the decision rule down before you see the second experiment's metric.
Watch out

Common mistakes

  • Comparing new model on train/test-v201 with baseline on test-v202.
  • Changing preprocessing + model + seed and crediting one factor.
  • Reporting best-of-5-seeds while the average is worse.
  • Peeking at results, then 'deciding' the rule retroactively.
Root cause analysis

Common issues

Error
Results flip when you add seeds 6-10
Root cause

Winner was within noise band (variance not captured)

Fix

Pre-commit a confidence band (e.g. ±2×std across seeds) as the adoption threshold.

Error
Baseline beats every treatment
Root cause

Baseline is actually strong; or splits drifted between runs

Fix

Rehash + rerun baseline on the exact current splits before despairing.

Error
p-values all non-significant
Root cause

Noise dominates effect or sample too small for the metric

Fix

Use paired tests + effect-size planning before running the experiment.

When things break

Debugging tips

  • A 'huge win' that only shows on one seed is probably luck — rerun with a wider seed range.
  • Score the baseline fresh in the same script as the treatment so nothing drifts.
  • Print the splits' target distribution per fold; an imbalanced fold invalidates the whole comparison.
  • Keep a changelog of treatments; chaos in the log usually predicts contradictory results.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The core of sound experiment design is:

2

Changing the validation split between baseline and treatment is acceptable if the test set is the same.

Go deeper

Official documentation

ML test score (Google)
docs