ML Experimentation
Intermediate 15 min read ML Experimentation

Reproducibility

Same inputs, same outputs — on purpose

Reproducibility means the same code + data + config + environment yields the same result. It's a discipline: pin code (git), data hashes, seeds, library versions and config — then a 'rerun' is a checkout + one command.

reproducibilityseedlockfilegitdata hashdeterminismDocker
Prerequisites: Experiment Tracking Environment & Tooling
Before we start

What you will learn

  • List the five reproducibility axes: code, data, config, environment, randomness
  • Pin seeds, dependency versions and data hashes
  • Build one-command reruns and determinism checks
Introduction

What is this?

Reproducibility makes a model's birth reproducible: identical git commit, dependency lockfile/image, config, dataset hash, and seeded randomness. GPU and library nondeterminism can still leak differences at the float level, so logs record run-to-run variation. Concretely: checkout commit, `docker run` the pinned image, `python train.py --config run.json`, and the same model-and-metrics come out.

Motivation

Why do you need it?

  • Bug fixes and audits require tracing a model back to exactly how it was made.
  • Comparisons are only meaningful inside a reproducible baseline.
  • Rolling back a production model = reproducing the previous build.
  • Multi-designer pipelines (engineers rotate) die without a common recipe.
Think about it

Mental model

Imagine

A recipe with exact measurements and oven settings

Think of

Reproducible training runs

Same cookbook version, same bag of flour (data hash), same oven profile (environment) and same measured out quantity of each ingredient (seeds/config) — and anyone in any kitchen can bake the identical cake. Skip the measurements once and the cake is a surprise every time.

Mechanics

How it works

  • Commit code + config; tag training code versions like releases.
  • Hash the dataset (SHA-256 of manifest) and record it in the run.
  • Pin environment: requirements lockfile/image or conda env export.
  • Seed Python/numpy/torch/model init — and LOG the seed values per run.
  • Verify: rerun from checkout in a fresh env and diff metrics against the recorded ones.

Five axes pinned before a reproducible run.

Code
Data
Config
Env
Randomness
Real world

In production

  • Model rollback: rebuild last quarter's model byte-for-byte if the new one regresses.
  • Audits and post-mortems: 'which exact build caused this?' answered in minutes.
  • Team onboarding: a new engineer reproduces any historical run from the tracker.
Guidelines

Best practices

  • Seed EVERYTHING and log the seeds (don't silently use default 0).
  • Version the data as a hash, not a folder name — folders get overwritten.
  • Containerize with a lockfile; bare requirements.txt is approximate.
  • Run a determinism check (2 identical runs) once per major dependency bump.
Watch out

Common mistakes

  • Seeding numpy but not torch (or scikit-learn opaque split random_state).
  • Dependency drift: 'ran last month' means nothing after a silent package upgrade.
  • Data folder mutated in place — hash changed, run claimed 'same data'.
  • Logging metrics but not the commit that produced them.
Root cause analysis

Common issues

Error
Same commit, different results
Root cause

Unpinned deps, data drift, or floating nondeterminism

Fix

Pin lockfile + data hash; then quantify residual variation across N identical runs — report that band.

Error
Seeds ignored
Root cause

A library resets its own RNG or seeds set after first use

Fix

Seed before library import/use and re-check by running twice and diffing.

Error
Reproduction fails on another machine
Root cause

Environment OS/GPU library mismatch despite lockfile

Fix

Ship a container image (not just requirements) and record its digest in the run.

When things break

Debugging tips

  • If in-commit reruns differ, compare environment hashes first, library versions second.
  • Diff two runs' metric histories; divergence halfway usually points at data ordering/streaming.
  • Freeze seeds via a config field, not scattered constants, so runs declare their own.
  • On GPU float mismatch, record tolerance-level determinism (same to N decimals) rather than requiring bit-equal.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The strongest way to pin an environment is:

2

Seeding only numpy is enough to guarantee a deterministic pipeline.

Go deeper

Official documentation

Reproducibility in scikit-learn
docs
PyTorch reproducibility notes
docs