Model Evaluation
Intermediate 17 min read Model Evaluation

ROC, AUC and PR Curves

Threshold-by-threshold truth, summarized without picking a cut line

ROC and PR curves show how a classifier behaves at every decision threshold. AUC summarizes rank quality but hides calibration; PR-AUC is the honest surface for rare-event problems.

ROCAUCprecision-recall curvePR-AUCthresholdTPRFPR
Prerequisites: Classification Metrics
Before we start

What you will learn

  • Construct ROC and PR curves from scores
  • Interpret AUC and PR-AUC and know what each summarizes
  • Choose the right curve family for imbalanced problems
Introduction

What is this?

Classifier probability outputs become labels only after a threshold: above it 'positive', below it 'negative'. Sweep that threshold from 1 to 0 and you trace a curve. The ROC curve plots TPR against FPR; the PR curve plots precision against recall. Their area under the curve — AUC (ROC-AUC) and average precision (PR-AUC) — gives a single, threshold-independent number summarizing how well the model ranks positives above negatives.

Motivation

Why do you need it?

  • A score of 0.7 for precision means nothing until you say which threshold produced it — curves show every choice at once.
  • ROC-AUC is robust to class ratio for ranking; PR-AUC tells the truth when positives are rare.
  • Comparing models by AUC avoids the trap of tuning each one to a different threshold then arguing over scalars.
  • The 'elbow' of a curve locates the sweet spot before you set a final threshold.
Think about it

Mental model

Imagine

A dimmer switch swept across its range

Think of

Sweeping the decision threshold

You don't have to pick bright or off — the curve shows you the entire trajectory of trade-offs as you slide the dimmer, and the area captures how good the range is overall.

Mechanics

How it works

  • The model outputs a score (probability) per example; sort the test set by score descending.
  • For each possible threshold, tally TPR = TP/(TP+FN) and FPR = FP/(FP+TN); plot them against each other (ROC).
  • The ROC curve starts at (0,0) and ends at (1,1); a random model is the diagonal, a perfect model hugs the top-left corner.
  • Area under the ROC (AUC) equals the probability that a random positive scores higher than a random negative.
  • For the PR curve, plot precision vs recall; the baseline is the class prevalence, and PR-AUC (average precision) summarizes it.

The same classifier as ROC and PR curves.

Score each example
Sweep threshold
ROC
PR
Area
Real world

In production

  • A ranking system promotes the model with the higher ROC-AUC because it only needs the best order, not a threshold.
  • A fraud team uses PR curves because positives are <0.5% — a PR curve shows whether the top-decile precision is usable.
  • A monitoring dashboard tracks PR-AUC drift instead of accuracy because it degrades visibly when fraud patterns shift.
Guidelines

Best practices

  • Use ROC-AUC when classes are roughly balanced and ranking matters; switch to PR when positives are rare.
  • Report the curve, not just the area — two models can share an AUC with very different curve shapes.
  • Compute curves on the same fixed test set; scores at different data distributions are not comparable.
  • Pick the point on the curve that matches your operating constraint (e.g., recall ≥ 0.9) before finalizing the threshold.
Watch out

Common mistakes

  • Quoting AUC=0.95 on a 99/1 problem where PR-AUC is 0.3 — the two tell a very different story.
  • Interpreting AUC as accuracy; 0.5 is random, 0.8 is decent, regardless of the 'out of 100' intuition.
  • Comparing PR-AUC across datasets with different prevalence — it shifts with the positive rate.
  • Using probabilities that were not calibrated on a curve analysis and reading them as literal frequencies.
Root cause analysis

Common issues

Error
ROC-AUC is 0.5 on training data
Root cause

No separation signal, or scores are inverted (higher score = more negative)

Fix

Check prediction direction and feature quality; try inverting the score if labels were flipped.

Error
PR-AUC looks great but live performance dies
Root cause

Test distribution was not representative or the model was tuned to the curve's lucky region

Fix

Cross-validate curves and re-estimate on slices; optimize in the operating region, not the whole area.

When things break

Debugging tips

  • If ROC looks like a staircase hugging the axes, your scores are too coarse (few unique probabilities).
  • A PR curve dropping instantly means the top-scoring examples are mostly false positives — inspect them.
  • AUC identical but business result worse? Compare the curves point-by-point in the operating region.
  • When AUC is 1.0, suspect leakage or duplicates between train and test before celebrating.
Interviews

Interview checkpoint

Practice

Hands-on lab

Trace ROC and PR Curves From Scratch
30 min · intermediate
In progress
Objectives
  • Compute TPR/FPR/precision/recall at every threshold by hand
  • Reproduce AUC and average precision without sklearn
  • Observe how class imbalance reshapes the PR curve but not the ROC shape

Trace ROC and PR Curves From Scratch

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A random classifier's expected ROC-AUC is:

2

PR-AUC and ROC-AUC degrade identically when the positive class becomes more rare.

Go deeper

Official documentation

sklearn ROC and PR curves
docs