ML Experimentation
Intermediate 15 min read ML Experimentation

Experiment Tracking

Every run, logged forever

Experiment tracking records every run: hyperparameters, metrics, artifacts and code version. Without it, 'which config beat the baseline?' is unknowable and every improvement is unverifiable. Tools: MLflow, W&B, custom SQLite + git.

experiment trackingMLflowrunmetricartifactbaselinereproducibility
Prerequisites: Evaluation Metrics Hyperparameter Optimization
Before we start

What you will learn

  • Log params, metrics, artifacts and environment per run
  • Compare runs and promote the verified best
  • Integrate tracking with git commit and dataset version
Introduction

What is this?

Experiment tracking persists one record per training run: hyperparameters, metrics (train/val/test), artifacts (weights, model, plots), config, git commit, data version and environment. That record makes comparisons honest, regression hunts possible, and promotion decisions revertible. A tracker can be a tool (MLflow, Weights & Biases) or a thin wrapper around a SQLite table + git tags — the discipline matters more than the tool.

Motivation

Why do you need it?

  • You cannot compare what you didn't log — 'it worked yesterday' isn't evidence.
  • Wasted compute: retraining to recall a forgotten best config is expensive.
  • Audit: compliance and debugging need to know exactly what produced a model.
  • Baselines become a first-class artifact, so improvements are provable.
Think about it

Mental model

Imagine

A lab notebook with a strict protocol

Think of

Experiment tracking

Every experiment gets a dated page: what went in, what happened, what came out, which equipment (env/seed). Months later you can open the winning page and reproduce it exactly — and you can prove to a reviewer that you tested the controls.

Mechanics

How it works

  • Wrap the run: start_run records start time, git commit and run name/hash.
  • Log hyperparameters (lr, n_estimators, data version) as params.
  • Log metrics at intervals (loss, auc) and at the end (final, best-epoch).
  • Store artifacts: model weights, plots, preds, eval reports with a path + hash.
  • Query: list/filter runs, compare tables, diff against baseline, promote via a declared best_commit.

One run's record feeds comparison and promotion.

Launch
Train
Log
Compare
Promote
Real world

In production

  • Hunt regressions: a metric dip ties to a specific git commit + data version.
  • Compliance: per-model lineage (code, data, config, metrics) on demand.
  • Auto-MLOps: winner of an HPO sweep auto-registers to a model registry.
Guidelines

Best practices

  • Put git commit + dataset hash in EVERY run; untagged runs are worthless.
  • Pick the promotion metric once (e.g. val-AUC) and log it consistently across runs.
  • Snapshot baseline runs first — before touching any hyperparameter.
  • Log raw numeric metrics, not just plots; plots compliment, they don't substitute.
Watch out

Common mistakes

  • Tuning parameters without an active run — the 'what did I change last night' trap.
  • Comparing runs with different data/validation splits and calling the diff a model win.
  • Logging final loss only, with no learning curve to inspect convergence.
  • Letting runs overwrite the same artifact path (no per-run directory).
Root cause analysis

Common issues

Error
Runs with identical name but different results
Root cause

No run hash or code-not-committed

Fix

Auto-tag with commit + dataset hash and reject runs missing those tags.

Error
Artifact path collision across runs
Root cause

Static file name in a shared dir

Fix

Per-run directory (run_id/) and content hashes in artifact names.

Error
Tracker down breaks training
Root cause

Tracking treated as critical path

Fix

Local-first logging with async/retry; a tracking failure must never fail training.

When things break

Debugging tips

  • Every run's loss curve should be immediately plottable — flat or spike curves localize bugs.
  • A metric that 'improves' but your check fails (NaN, negative) is tracking pollution — validate before logging.
  • Diff one param at a time vs baseline: attribution only survives single-variable changes.
  • Check run count per day; runs you can't account for are usually config mistakes.
Interviews

Interview checkpoint

Practice

Hands-on lab

Track Runs Like a Pro
30 min · intermediate
In progress
Objectives
  • Build a minimal run-logging store
  • Log params, metrics and hash per run
  • Query and rank runs to pick a winner

Track Runs Like a Pro

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which is NOT a reason to log artifacts per run?

2

A valid comparison needs the same validation split across runs.

Go deeper

Official documentation

MLflow tracking docs
docs
Weights & Biases docs
docs