Evaluation Metrics That Matter
The scoreboard of supervised ML
Accuracy hides more than it shows. Precision/recall, F1, ROC-AUC, calibration and per-segment breakdowns turn 'the model is good' into 'the model is good for this decision under these conditions'.
What you will learn
- Explain precision, recall, F1 and when each matters
- Read confusion matrices and ROC curves
- Choose segment-level metrics over aggregate averages
What is this?
Evaluation metrics convert model output into decision-relevant numbers. For classification: a confusion matrix, then precision (of what you flagged, how much was right), recall (of what was real, how much you caught) and F1 (their harmonic balance). ROC-AUC ranks truth across thresholds; calibration says whether probabilities are honest. The right metric follows the cost of being wrong.
Why do you need it?
- Accuracy on imbalanced data is a lie — 99% 'correct' by never predicting the rare class.
- Precision vs recall trades which error hurts: false alarms (spam filters, guardrails) vs missed catches (fraud, safety).
- AUC tells you ranking quality; threshold tells you the operating point.
- Aggregate metrics hide which user segments or classes the model quietly fails on.
Mental model
A fishing net
Precision vs recall
A net catches fish (recall) but also pulls weed (low precision); a narrow net catches only keepers (precision) but misses most (low recall). You tune the mesh to the day's goal.
How it works
- Confusion matrix: four cells — TP, FP, TN, FN — everything else is derived from them.
- Precision = TP/(TP+FP); recall = TP/(TP+FN); F1 = harmonic mean, resists single-metric gaming.
- ROC-AUC summarizes ranking quality across thresholds; a threshold later fixes the operating point.
- Calibration (ECE/binning) checks whether 0.9 probability means ~90% of the time.
- Segment metrics: by class, region, cohort — the model may win overall and lose your hardest users.
From raw predictions to decision metrics.
In production
- A safety classifier gates on precision: block-only-what-you're-sure, accepting missed flag as cheap.
- A fraud model optimizes recall within a rollout where catch-more a small FP cost is cheap.
- An eval dashboard reports F1 overall plus per-region F1, surfacing silent regional degrades.
Best practices
- Derive metrics from the decision's error cost, not from convenience.
- Report the confusion matrix with any headline metric — averages hide composition.
- Choose a threshold on validation data and re-verify on a truly reserved set.
- Track metrics per segment and over time to catch drift before it hears about it.
Common mistakes
- Reporting accuracy on imbalanced data as the 'model quality' headline.
- Tuning on F1 while the product only cares about precision at a fixed recall.
- Threshold at 0.5 without considering error costs.
- Comparing models on different eval sets and declaring a winner.
Common issues
High accuracy, zero detections in the rare classImbalanced classes and un-stratified metrics
Use per-class precision/recall and rebalance training or rewards.
Calibrated offline, overconfident liveServing distribution or features differ from eval
Monitor calibration drift live and re-calibrate on refreshed data.
Improvement shows overall, disappears per segmentGains concentrated in a common segment
Grid metrics by segment and require per-segment gates.
Debugging tips
- F1 high but the hard cohort failing — break metrics by segment before touching the model.
- AUC great, deployment poor — your threshold or calibration is off, not the ranking.
- Recall collapses after a data refresh — re-verify class priors and label consistency.
Interview checkpoint
Hands-on lab
- Build predictions and a confusion matrix from scratch
- Compute precision, recall, F1 and ROC-AUC
- Show why accuracy lies on imbalanced data
Score a Classifier Honestly
25 min · intermediate
Quick quiz
Precision answers which question?
F1 is the harmonic mean of precision and recall.