Experiment Tracking
Beginner 12 min read Experiment Tracking

Why Track Experiments

Memory is not a storage system

Without tracking, every experiment is a guess in the dark: what dataset, what code, what parameters produced that score? Tracking turns model development into an auditable, comparable process.

experiment trackingMLflowreproducibilitymetricsparametersruns
Prerequisites: From Data to Model
Before we start

What you will learn

  • Explain why raw scores without context are worthless
  • List the core pieces a tracked run must record
  • Describe how tracking changes team collaboration
Introduction

What is this?

Experiment tracking is the practice of recording every training run's parameters, code version, data version, metrics and artifacts so the run can be replayed, compared and defended. A tracked run answers 'what produced this number' — untracked runs answer nothing.

Motivation

Why do you need it?

  • ML experimentation is iterative: the 40th model only makes sense next to runs 1–39.
  • An impressive metric without its recipe is a rumor; auditors and stakeholders need the recipe.
  • Tracking enables collaboration — another data scientist can build on your run instead of redoing it.
  • Regression hunting requires before/after comparisons; tracking is the diff tool.
Think about it

Mental model

Imagine

A lab notebook

Think of

Experiment tracking

A chemist records reagents, conditions and results because a result without a method is unusable. A tracked ML run is the same notebook — machine-readable, comparable and shared.

Mechanics

How it works

  • Each training invocation creates a 'run' with a unique ID.
  • Parameters (the inputs you set) are logged as key-values.
  • Metrics (the outputs you evaluate) are logged during and after training.
  • Artifacts — model files, plots, datasets previews — are attached to the run.
  • Environment info (git commit, library versions, data hash) pins the context.
  • Runs are indexed, searchable and sortable for comparison.

Anatomy of a tracked run.

Training starts
Log params
Log metrics
Attach artifacts
Pin context
Run recorded
Real world

In production

  • A DS team compares 120 nightly runs across 6 model families by searching metrics and sorting winners.
  • A compliance review points at production model 'v12' and the tracker reproduces its exact training recipe.
  • A candidate model regresses in staging; the team diffs its tracked features against the champion.
Guidelines

Best practices

  • Track from your very first run — retrofitting tracking is nearly impossible.
  • Log parameter defaults explicitly, not just overrides.
  • Make run names meaningful (model-family/data-window), not 'final_v2'.
  • Add tags for ownership, experiment line and hypothesis.
Watch out

Common mistakes

  • Printing numbers to console instead of logging them.
  • Naming runs 'final', 'final_v2', 'final_really'.
  • Logging parameters after the run finished, corrupting the recipe.
  • No data version in the run, so the recipe is incomplete.
When things break

Debugging tips

  • Can't reproduce a good run? Rebuild the recipe from its tracked params, commit and data hash.
  • Two runs 'identical' but different scores? Diff the environment and data versions in the tracker.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Recording only the final accuracy is enough to reproduce a model.

2

Which of these belongs in a run's recorded context?

Go deeper

Official documentation

MLflow Tracking overview
docs
Weights & Biases Experiments
docs