Model Evaluation
Intermediate 16 min read Model Evaluation

Classification Metrics

Beyond accuracy: seeing exactly which mistakes matter

Accuracy hides class-imbalance failures. Precision, recall, F1 and the confusion matrix reveal exactly which errors a classifier makes and let you weight them by their real cost.

confusion matrixprecisionrecallF1imbalanceclassificationTPRFPR
Prerequisites: Metrics Fundamentals
Before we start

What you will learn

  • Read and build a confusion matrix
  • Explain precision, recall and F1 and their trade-off
  • Choose the metric pair that matches your business risk
Introduction

What is this?

Classification metrics describe a model's decisions against the truth using four counts: true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). From these you derive precision (of everything predicted positive, how much is real), recall (of all real positives, how much was caught), and the harmonic-mean F1. Accuracy is just (TP+TN)/(total) and looks great even when one class is never predicted.

Motivation

Why do you need it?

  • Accuracy is useless on rare-event problems — a 99% accurate fraud model may catch zero fraud.
  • Precision and recall map directly to costs: false positives annoy; false negatives lose money or lives.
  • The confusion matrix shows the failure structure (which classes get confused) that a single scalar hides.
  • Imbalanced problems dominate real ML: fraud, churn, disease, defects.
Think about it

Mental model

Imagine

A fire alarm

Think of

A classifier

A good alarm should never miss a fire (high recall) but also shouldn't cry wolf on toast (high precision). The right balance depends on whether a missed fire or a false alarm is more expensive in your building.

Mechanics

How it works

  • You collect a labeled set of examples and predict each one with the classifier.
  • You tally TP/FP/TN/FN by comparing predictions to labels — an example predicts positive and is truly positive, etc.
  • Precision = TP/(TP+FP): how trustworthy a positive prediction is.
  • Recall/sensitivity/TPR = TP/(TP+FN): how much of the actual positive set was caught.
  • You may also compute specificity = TN/(TN+FP) and F1 = 2*P*R/(P+R), the balanced harmonic mean.

Confusion matrix to derived metrics.

Predictions
Tally
Precision
Recall
Combine
Hands-on

Examples

Compute metrics from a 100-row sample

Compute metrics from a 100-row sample
python3 -c "
tp, fp, tn, fn = 30, 10, 45, 15
prec = tp / (tp + fp)
rec = tp / (tp + fn)
f1 = 2 * (prec * rec) / (prec + rec)
print('precision', round(prec,3), 'recall', round(rec,3), 'F1', round(f1,3))
print('accuracy', round((tp+tn)/100,3), 'false alarms', fp, 'misses', fn)
"

Accuracy is 0.75, which flatters a system that missed 15 real positives and raised 10 false alarms. Precision and recall isolate the two costs so a stakeholder can weigh them separately.

Real world

In production

  • A medical-screening model gates on recall ≥ 0.98 because missing a case is worse than retesting a healthy patient.
  • An ad-click model optimizes precision at a fixed budget — it must not waste impressions on bad users.
  • Customer-support routing uses a per-class confusion matrix to find which ticket types are systematically misrouted.
Guidelines

Best practices

  • Always flatten the confusion matrix before trusting any derived scalar.
  • Report metrics per class, especially for multi-class; the macro-average treats every class equally.
  • Set the metric from the cost ratio: if a false negative costs 10x a false positive, weight accordingly.
  • State the positive class explicitly — swapping positive/negative mid-review flips every number.
Watch out

Common mistakes

  • Quoting accuracy on imbalanced data without showing the per-class break-down.
  • Optimizing F1 blindly when one error type costs 100x the other.
  • Averaging precision and recall arithmetically (mean) instead of harmonically (F1).
  • Forgetting the test set must reflect the same class distribution as production.
Root cause analysis

Common issues

Error
Recall = 1.0 and precision = 0.2 on fraud
Root cause

Model predicts 'fraud' for almost everything

Fix

Move the decision threshold up or rebalance training; then recheck using a threshold-independent curve like PR-AUC.

Error
F1 improved but revenue dropped
Root cause

F1 treated precision and recall equally; the business weighs misses harder

Fix

Use F_beta or directly count expected cost per confusion cell.

When things break

Debugging tips

  • Print the confusion matrix first — a model predicting one class everywhere shows a single hot row.
  • If precision and recall both fall, check the model output distribution vs the labels, not the metric code.
  • For multi-class, examine the off-diagonal cells to see which real classes collapse together.
  • Recheck the positive class encoding when recall suddenly becomes 1.0.
Interviews

Interview checkpoint

Practice

Hands-on lab

Read a Confusion Matrix Like a Pro
25 min · intermediate
In progress
Objectives
  • Train a quick classifier on an imbalanced synthetic dataset
  • Derive precision, recall, F1 and specificity from the confusion matrix
  • Show how accuracy hides the failure to catch rare positives

Read a Confusion Matrix Like a Pro

25 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A model predicts fraud 5 times: 3 are real fraud, 2 are mistakes. Its precision is:

2

A model with perfect recall always has high precision.

Go deeper

Official documentation

sklearn classification metrics
docs