ML Fundamentals
Intermediate 14 min read ML Fundamentals

Metrics and Evaluation

Pick the number the business cares about

Accuracy lies on imbalanced problems; precision/recall/F1, AUC-ROC/PR, MAE/RMSE/R2 and log-loss each tell a different story — pick the metric the business actually cares about.

metricsaccuracyprecisionrecallF1AUCROCMAERMSER2confusion matrix
Prerequisites: Cross-Validation
Before we start

What you will learn

  • Read a confusion matrix and define TP/FP/FN/TN, precision, recall, accuracy, F1
  • Compare AUC-ROC vs PR AUC and choose by class balance
  • Choose regression metrics: MAE vs RMSE vs R2 and their failure modes
Introduction

What is this?

Metrics quantify model quality in one comparable number. Classification: accuracy (right/total), precision (of predicted positives, how many real), recall (of real positives, how many caught), F1 (harmonic mean), ROC-AUC (rank quality across thresholds), PR-AUC (precision/recall balance for rare positives), log-loss (probabilistic). Regression: MAE (median-ish error), RMSE (penalizes large errors), R2 (variance explained), MAPE (percent error — careful on small targets).

Motivation

Why do you need it?

  • On 99% majority-class data, an accuracy-optimized model can be a useless constant predictor.
  • Fraud wants recall (catch thieves) but alert overflow costs money — precision matters equally; only a pair tells the truth.
  • RMSE vs MAE decides whether big misses get punished; the business cost of a 10x error decides which one.
  • AUC rank quality is robust to thresholds; PR-AUC is honest when positives are rare.
Think about it

Mental model

Imagine

A fire detector's sensitivity vs its false alarms

Think of

Precision vs recall

A very sensitive detector catches every fire (high recall) but screams at toast (low precision); a cautious one never bothers you (high precision) but misses fires. The right tradeoff depends on the cost of each miss.

Mechanics

How it works

  • Classification: build the confusion matrix (TP/FP/FN/TN) first; read rates off it.
  • Precision = TP/(TP+FP); recall = TP/(TP+FN); accuracy = (TP+TN)/total; F1 = 2PR/(P+R).
  • ROC-AUC: threshold sweep of TPR vs FPR; PR-AUC: precision vs recall as threshold moves.
  • Select threshold by business cost: minimize cost = (FP*cost_fp + FN*cost_fn) rather than default 0.5.
  • Regression: look at residual spread; RMSE punishes a 50 error more than 5x a 10 error, MAE treats them evenly.

Classification metric map.

Confusion
Rates
Rank
Threshold
Hands-on

Examples

Metrics on an imbalanced fraud set

Metrics on an imbalanced fraud set
import numpy as np
from sklearn.metrics import (
    confusion_matrix, precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score, log_loss,
)
 
rng = np.random.RandomState(0)
y = np.array([0]*970 + [1]*30)
prob = y.astype(float).copy()
prob[y == 1] = 0.9 + rng.random(30) * 0.1
prob[y == 0] = rng.random(970) * 0.5
 
pred = (prob > 0.5).astype(int)
print(confusion_matrix(y, pred))
print("precision", round(precision_score(y, pred), 3),
      "recall", round(recall_score(y, pred), 3),
      "f1", round(f1_score(y, pred), 3))
print("ROC-AUC", round(roc_auc_score(y, prob), 3),
      "PR-AUC", round(average_precision_score(y, prob), 3),
      "logloss", round(log_loss(y, prob), 3))

Accuracy would look ~97% yet the detector's true usefulness is visible only through precision/recall and the rank-based AUCs, which do not cheerlead the majority class.

Real world

In production

  • Fraud ops: optimize F1 *at a chosen threshold* plus cost-weighted alert value, not accuracy.
  • Retail demand: MAPE on low-volume SKUs explodes — switch to weighted MAPE or RMSE in log space.
  • Recommendation ranking: AUC/NDCG on the ranking, and click-through as the business meter.
Guidelines

Best practices

  • Report the business metric FIRST and the model metric second; on imbalanced data prefer PR-AUC over ROC-AUC; state the threshold you deployed and why; for regression pair RMSE (outlier penalty) with MAE (typical error).
Watch out

Common mistakes

  • Bragging an accuracy number on a 99/1 target; using a fixed 0.5 threshold when a 0.2 threshold is cheaper; choosing RMSE when one 50x outlier is, in fact, a data bug; interpreting R2 as 'percent correct'.
When things break

Debugging tips

  • Print the full confusion matrix, not just scalar metrics.
  • Plot predicted-prob distribution vs label — a gappy or miscalibrated histogram explains weird F1s.
  • Rerun metrics at multiple thresholds to see how robust the win is.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Recall measures:

2

Which metric reads 'how many true positives caught per false alert raised'?

Go deeper

Official documentation

sklearn.metrics module
docs
Classification: precision and recall | Google
video
Precision and recall | Wikipedia
docs