Experiment Tracking
Advanced 14 min read Experiment Tracking

Team Practices for Experiment Tracking

The tracker is a team interface, not a personal diary

Shared tracking changes how teams collaborate: naming conventions, tags as contracts, governance of the tracker, and review flows that keep the platform trustworthy as models multiply.

ML team collaborationtracking governanceexperiment hygienetagsnaming conventions
Prerequisites: Comparing and Selecting Models
Before we start

What you will learn

  • Design experiment and naming conventions for a team
  • Use tags as machine-readable contracts
  • Enforce hygiene so the tracker stays searchable at scale
Introduction

What is this?

Team experiment practices turn a shared tracker into reliable shared memory: everybody logs the same way, tags carry meaning automation can use, naming conventions make runs searchable, and governance (permissions, retention, review) keeps thousands of runs trustworthy.

Motivation

Why do you need it?

  • A tracker full of 'final2' runs and missing tags is just a more searchable notebook.
  • Consistent tags enable automation: promotion pipelines read tags, not human memory.
  • Hygiene scales: 10 people × 50 runs/week becomes unsearchable without conventions.
  • Governance prevents someone's debug garbage from confusing promotion reviews.
Think about it

Mental model

Imagine

A shared library's cataloging rules

Think of

Experiment tracking conventions

A library where everyone shelves books wherever they like is useless. Dewey numbers (conventions), a catalog (tags) and a lending policy (permissions) make it usable by thousands.

Mechanics

How it works

  • Teams adopt a run naming scheme: {problem}/{model_family}/{experiment_line}.
  • A minimum tag set is enforced: owner, role (candidate/champion/debug), data-window, model-family.
  • Experiments are named by problem space, and archived as they retire.
  • Permissions restrict who can transition model stages or delete experiments.
  • A review workflow reads tags to surface promotion-worthy candidates.
  • Retention policies cap artifact and metric growth so search stays fast.

Team practices around the shared tracker.

Conventions
Standard logging
Tags as contract
Review flow
Governance
Real world

In production

  • A 12-engineer ML team runs 400 experiments/month; a reporter queries champion/candidate tags for the monthly review.
  • An on-call engineer answers 'what changed' by diffing tags across champion runs.
  • A promotion pipeline only considers runs tagged role=candidate with owner set.
Guidelines

Best practices

  • Codify conventions in the repo's README and enforce with a lint script.
  • Treat the tag schema as an API — change it with review and migration.
  • Route promotion automation through tags, never free-text run names.
  • Review the tracker quarterly: archive dead experiments, drop unused artifact stores.
Watch out

Common mistakes

  • A tag-free culture where 'we know what this is' — until someone leaves.
  • Run names that collide across people ('final' exists 40 times).
  • Deleting experiments to 'clean up', destroying audit history.
  • No ownership metadata, so nobody answers for a run in production.
When things break

Debugging tips

  • Can't find a run? Search by tags first, then by parameter ranges.
  • Automation promoting nothing? Check your tag schema matches the pipeline's filter.
  • Tracker slow? Partition experiments and archive; metrics curves are usually the growth culprit.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What is the strongest reason to enforce a minimum tag set on every run?

2

Deleting old experiments is always a safe way to keep the tracker fast.

Go deeper

Official documentation

MLflow — experiments and tags
docs
MLflow registry permissions model
docs