Comparing and Selecting Models
Pick the champion, not the loudest metric
Winning an experiment isn't the highest accuracy — it's the model that clears the gates: metric targets, business cost, robustness and simplicity. Comparison is where tracking pays off.
What you will learn
- Define the gate criteria for selecting a production model
- Compare runs on aggregate and by business-impact metrics
- Involve robustness and deployment cost in the decision
What is this?
Model selection turns tracked runs into a decision: which candidate clears accuracy targets, meets latency/cost budgets, and behaves robustly across slices, then advances to registration. It is a gate-review process, not 'pick the best number'.
Why do you need it?
- Global accuracy hides failure on the segments that matter most (minorities, rare cases).
- A slightly worse model that is 5× faster or fully explainable is often the right production choice.
- Robustness to data shift matters more than a point metric that won't hold in production.
- Formal gates make the choice defensible and auditable.
Mental model
Hiring for a role, not a talent show
Model selection
The talent show crowns the most impressive single act; hiring checks fit, reliability under pressure and whether the person will last. Metrics are the audition, gates are the interview, robustness is the reference check.
How it works
- Collect candidate runs from the tracker with their parameter sets and metric curves.
- Apply the selection criteria: primary metric target, secondary business metrics, slice-level checks.
- Price the model: latency, memory, serving cost, retraining cost.
- Assess robustness: behavior across slices, sensitivity to input perturbations, degradation under drift.
- Write the selection memo: winner, runners-up, evidence, and the owner sign-off.
- Register the winner in the model registry to begin staged rollout.
From tracked candidates to a registered champion.
In production
- A healthcare triage team picks the model with best sensitivity on the smallest risk segment, not highest overall AUC.
- A real-time fraud team accepts 0.1% worse recall for a 40ms latency win that fits the transaction budget.
- A bank requires the chosen model to stay within 2% of baseline accuracy under simulated feature corruption.
Best practices
- Write the selection gates before the results arrive — avoid moving goalposts.
- Compare runs on the same fixed eval version only.
- Check slices: per-class metrics, small segments, geographic splits.
- Make a written selection memo; the registry entry references it.
Common mistakes
- Choosing the global-best metric model without checking minority-class performance.
- Ignoring serving constraints until the chosen model can't meet latency.
- Comparing runs evaluated on different data versions.
- Re-selecting with different gates after seeing results — that's p-hacking your own process.
Debugging tips
- The 'best' model fails in production? Run your perturbation/slice battery before promotion, not after.
- Selection and eval disagree? Confirm both used the same data version and metric definitions.
- Candidates tied? Add cost and complexity as tiebreakers, and start a shadow run.
Interview checkpoint
Hands-on lab
- Generate a small comparison table from tracked runs
- Implement a written selection gate in Python
- Select a champion and generate a selection memo
Run Comparison with a Gate
25 min · intermediate
Quick quiz
Model A: overall F1 0.91, F1 on minority class 0.4. Model B: overall F1 0.88, F1 on minority class 0.78. Which should likely advance?
Selection gates should be written down before looking at the leaderboard.