Dataset Reproducibility
Re-run anything, any time, from the same inputs
Reproducibility is the property that a model's training pipeline can be re-run against the same inputs and produce the same outcome. It's a discipline: pins, seeds, immutability, and replay.
What you will learn
- Pin data, code and environment for a reproducible run
- Diagnose nondeterminism in training
- Design replay verification for critical models
What is this?
Dataset reproducibility is the end-to-end guarantee that re-running the pipeline with the same pinned data, code and environment reproduces the same artifacts — or that any divergence is understood. It sits on data versioning but requires more: deterministic ordering, fixed seeds, and a recorded environment recipe.
Why do you need it?
- Auditors and reviewers must be able to replay a claimed result.
- Debugging a regression needs a 'same inputs' control, not a cold re-train.
- Automated retraining becomes safe only when runs are causally repeatable.
- Repro separates data problems from model problems cleanly.
Mental model
Baking an exact cake from a recorded recipe
Dataset reproducibility
Same ingredients (data), same oven settings (config), same technique (code) — the cake comes out the same. Change a single ingredient or gram and the result shifts. Reproducibility is recording the full recipe, not just the cake photo.
How it works
- The full recipe is captured: data version hashes, code commit, config, seeds, dependency lockfile.
- Inputs are immutable: data and artifacts never change in place.
- Determinism is engineered: fixed data ordering, fixed seeds, pinned random states.
- A replay run verifies: re-run with pinned inputs and compare hashes of outputs.
- Divergences (e.g. non-deterministic GPU kernels) are documented and accepted with thresholds.
The reproducibility recipe and its verification loop.
In production
- A finance model's registry entry carries the full recipe; a compliance replay reproduces the exact held-out metrics.
- A retraining pipeline fails its own replay check 2 runs in a row, prompting a determinism investigation.
- A research bug is traced to unshuffled data — the reproducible pipeline's hash diff identified it in minutes.
Best practices
- Version the environment: lockfiles inside the artifact picture.
- Fix seeds and data ordering at the pipeline level, not in scattered code.
- Replay on critical promotions as a gate — cheap relative to a bad model in production.
- Parameterize nondeterminism (GPU kernels, distributed collectives) and record expected tolerance.
Common mistakes
- Pinning data but not code, or code but not data — half a recipe.
- Systematic random() calls that ignore seeds and shuffle every run.
- Reading files in directory API order, which varies across filesystems.
- Treating 'reproducible' as binary when real systems accept bounded divergence.
Debugging tips
- Two runs differ? Hash every intermediate artifact to localize the first divergence.
- Floating-point drift? Pin hardware-seeded ops or document tolerance per op.
- Replay 'impossible' to reproduce? Hunt for hidden randomness: env vars, time, network, load order.
Interview checkpoint
Hands-on lab
- Pin seeds and data ordering for a repeatable split
- Prove two runs produce identical outputs
- Bisect a divergence to its source
Make a Run Deterministic
25 min · advanced
Quick quiz
Which pair is required for a complete reproducible recipe?
Bit-for-bit determinism is always achievable in deep learning training.