Model Evaluation
Intermediate 14 min read Model Evaluation

Threshold Selection

Where on the curve you decide to act

A classifier's business value lives at a single operating point on its score curve. Choosing the threshold from cost matrices, expected profit, or constraints turns a raw model into a decision system.

thresholdoperating pointcost matrixF_betadecision boundaryexpected value
Prerequisites: Classification Metrics
Before we start

What you will learn

  • Explain why the default 0.5 threshold is rarely right
  • Pick thresholds from cost matrices and profit lines
  • Evaluate threshold choice without overfitting to the test set
Introduction

What is this?

Every probabilistic classifier ships with a decision rule: act when score ≥ threshold, otherwise don't. The generic 0.5 boundary assumes positives and negatives are equally likely and equally costly, which almost never holds in production. Threshold selection finds the operating point — the specific precision/recall trade-off — that maximizes expected value under your real costs and constraints.

Motivation

Why do you need it?

  • At 0.5, a rare-event model usually does nothing; the useful operation lives at 0.1 or 0.02.
  • Precision and recall can both be 'good' while the chosen point loses money.
  • The threshold, not the AUC, is what the consumer actually experiences.
  • Setting it with cost math turns a vague trade-off into a budget decision.
Think about it

Mental model

Imagine

Picking a trading stop-loss

Think of

Choosing a decision threshold

A stop-loss is only 'correct' given your downside tolerance and upside opportunity — the same is true of a threshold, which encodes how you trade misses against false alarms.

Mechanics

How it works

  • Assign a cost to each confusion cell: c_FN (missed positive), c_FP (false alarm), maybe c_TP, c_TN.
  • For any threshold, expected cost per decision = P(FN)·c_FN + P(FP)·c_FP (normalized by prevalence).
  • Alternatively, work from a profit line: expected value = TP_gain·recall·prevalence − FP_cost·fpr·(1−prevalence).
  • Sweep thresholds, compute the expected value at each, and pick the argmin cost / argmax profit.
  • Fall back to a constraint: 'recall must be ≥ 0.95' → pick the highest-precision threshold that still meets it.

Choosing an operating point on the precision/recall curve.

Costs
Score curve
Expected value
Operating point
Verify
Hands-on

Examples

Threshold from a cost ratio

Threshold from a cost ratio
python3 -c "
import numpy as np
# costs (arbitrary units): miss a fraud = 100, false alarm = 1
c_fn, c_fp = 100, 1
# example decision thresholds with their observed rates (from ROC)
rates = [(0.05, 0.60), (0.10, 0.80), (0.20, 0.75)]  # (fpr, tpr)
prev = 0.01
for thr, (fpr, tpr) in rates:
    cost = (1-tpr)*c_fn*prev + fpr*c_fp*(1-prev)
    print(f'threshold {thr}: expected cost {cost:.4f}')
"

Because a miss costs 100× a false alarm, the threshold with the highest recall wins — the cost ratio moves you down the curve, not 0.5.

Real world

In production

  • A fraud team moves the threshold up when a false positive costs a customer-relationship penalty and down when attackers get louder.
  • A medical screener declares 'recall ≥ 0.97 or we don't ship' — threshold chosen to meet the constraint.
  • An ads platform sets the threshold by CPA target: only show predictions whose precision clears the cost-per-acquisition bar.
Guidelines

Best practices

  • Ground the cost matrix in real numbers from finance, not guesses in a meeting.
  • Validate the chosen threshold on a separate split; choosing it on the test set overfits your operating point.
  • Re-evaluate the threshold when prevalence or business costs drift — not just when the model changes.
  • Document threshold + rationale in the model card; a threshold with no owner is a silent risk.
Watch out

Common mistakes

  • Assuming 0.5 is neutral — it implies equality that almost never holds.
  • Tuning the threshold until test precision looks perfect (test-overfit operating point).
  • Picking threshold on validation, reporting curve on test, then shipping the validation threshold.
  • Ignoring that scores drift with data shift, so the old threshold's meaning shifts even if the value stays fixed.
Root cause analysis

Common issues

Error
The cost-Optimal threshold flags half the population
Root cause

c_FN/c_FP ratio extreme or prevalence mis-estimated

Fix

Re-derive prevalence from production data and sanity-check cost inputs with the business owner.

Error
Live results differ from the test-set operating point
Root cause

Score distribution shifted after deployment

Fix

Track score distribution drift and re-pick the threshold on recent data regularly.

When things break

Debugging tips

  • Plot expected value vs threshold — if it's flat, your cost inputs may be wrong or the model has no edge.
  • Recheck which class is 'positive'; flipping it silently mirrors the entire threshold discussion.
  • When a production threshold 'stops working', recompute the current curve before blaming the model.
  • Simulate the decision outcome on a historical week before committing a new operating point.
Interviews

Interview checkpoint

Practice

Hands-on lab

Pick the Profit-Maximizing Threshold
30 min · intermediate
In progress
Objectives
  • Compute expected profit at every threshold from a cost matrix
  • Show that the optimal operating point is far from 0.5
  • Validate the chosen threshold on a fresh split

Pick the Profit-Maximizing Threshold

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A model at threshold 0.5 misses 90% of fraud. The best next step is:

2

The threshold should be revisited when the cost of a false positive changes.

Go deeper

Official documentation

sklearn threshold tuning
docs