Model Evaluation
Beginner 12 min read Model Evaluation

Metrics Fundamentals

One number that tells a story — if you pick the right one

A metric converts model behavior into a single number you can act on. Choosing the right metric matters more than most modeling work — it decides which model wins and what the business sees.

model evaluationmetricslosserror metricsevaluation design
Prerequisites: Regression
Before we start

What you will learn

  • Explain why metric choice changes which model gets shipped
  • Match metric families to task type (regression, classification, ranking)
  • Spot when a metric is being gamed and needs a sharper cousin
Introduction

What is this?

A metric is a scalar function that scores how well a model's predictions match reality on a fixed dataset. Metrics distill hundreds of thousands of errors into one comparable number, but they also encode an assumption about what 'good' means — so the same model can rank first on one metric and last on another. Choosing the metric is a product decision, not a statistics footnote.

Motivation

Why do you need it?

  • Model A and B both 'work', but only a metric can say which one is better and by how much.
  • Metrics expose failure modes you would miss visually — a 95% accuracy model can still be useless for rare diseases.
  • The right metric aligns optimization with the business objective (money, safety, engagement).
  • A poorly chosen metric silently optimizes the wrong thing for months.
Think about it

Mental model

Imagine

A thermometer vs a barometer

Think of

A metric vs a model

Neither instrument tells you everything; you pick the one that measures what you actually care about. Metrics are the same — you choose the instrument that matches the risk you are worried about, not the one that looks most impressive.

Mechanics

How it works

  • You hold out data the model never trained on (validation/test) so the metric measures generalization, not memory.
  • You compute an error between each prediction y_hat and true value y using a per-row error function.
  • You aggregate per-row errors into a scalar — mean (sensitive to outliers), sum, median, or a ratio.
  • You compare the scalar across models or against a baseline (majority class, mean predictor, previous version).
  • You sanity-check the metric against a small sample you can verify by hand before trusting it at scale.

From predictions to a decision-making number.

Predictions
Errors
Aggregate
Compare
Decision
Real world

In production

  • The ads team ranks models by expected revenue lift — not accuracy — because a 0.5% accuracy gain that loses $2k/day is a loss.
  • A lending model is gated on recall of default at a fixed threshold, so a metrics drift alert fires when that recall drops.
  • A ranking team evaluates with nDCG@10 because users only see the top ten results.
Guidelines

Best practices

  • Define the primary metric from the business objective before training begins.
  • Report at least two metrics: the decision metric plus a diagnostic (e.g., accuracy + calibration).
  • Track metrics on the same untouched test set across every experiment for apples-to-apples comparison.
  • Log metric definitions with experiments — a metric that changes meaning is a metric that lies.
Watch out

Common mistakes

  • Letting the primary metric be whichever number looks best at review time.
  • Skimming the average and ignoring where errors concentrate (segments, tails).
  • Comparing models trained on different data/feature mixes as if the metric were comparable.
  • Reporting test metrics computed after tuning on that set — an inflated, optimistic number.
Root cause analysis

Common issues

Error
Accuracy says 98% but the product is failing
Root cause

Class imbalance — the model predicts the majority class and is rewarded

Fix

Use a metric that penalizes minority-class misses (recall, F1, PR-AUC) and inspect the confusion matrix.

Error
Same model, two metrics, opposite winners
Root cause

Metrics weight errors differently (RMSE punishes big errors, MAE treats all equally)

Fix

Choose the single metric whose error weighting matches your cost structure.

Error
Validation metric is higher every day even though nothing changed
Root cause

Metric code or data pipeline changed without being logged

Fix

Hash the eval script + dataset version into every experiment record.

When things break

Debugging tips

  • Recompute the metric on a 50-row sample by hand to catch sign and aggregation bugs.
  • Plot the error distribution, not just the mean — a perfect RMSE can hide a bimodal disaster.
  • If two metrics disagree, figure out which one is sensitive to the errors you actually care about.
  • Log the exact version of the evaluation script; metric code changes silently invalidate history.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which describes the main job of an evaluation metric?

2

A metric's assumptions should match the business cost of errors.

Go deeper

Official documentation

scikit-learn metrics
docs