Intermediate 13 min read Classification

Threshold Tuning and Cost

0.5 is a suggestion, not a rule

The 0.5 threshold is rarely right: sweep probabilities against the cost of false positives and false negatives to find the operating point that minimizes real money lost.

thresholdROC curvecost matrixoperating pointfalse positive costtradeoff

Prerequisites: Class Imbalance

Before we start

What you will learn

Sweep thresholds to build PR/ROC operating curves
Set a threshold from a stated cost matrix
Explain the threshold-to-business-cost mapping

Introduction

What is this?

Classifiers rank rows by probability; the threshold at which you call them positive is a business decision. Choosing a price: set the predicted-positive bar where the expected cost (FN x cost_fn + FP x cost_fp) is minimized — or where recall@top-k suits a queue. The ROC/PR curve over all thresholds shows every operating point and its trade.

Motivation

Why do you need it?

The same model optimized for accuracy vs recall is two different products.
Cost asymmetry (a missed fraud = $1k, a false alert = $2) must be in the threshold, not the loss.
Thresholds tune a deployed model at ANY time without retraining — your live cost dial.
Interview-standard skill: reading an operating point off a curve by cost.

Think about it

Mental model

Imagine

Setting the sensitivity of a smoke alarm

Think of

Choosing the classification threshold

Sensibility too high = toast alarms; too low = missed smoke. You pick the setting from the actual cost of burnt toast vs burnt house, not from 'default sensitivity'.

Mechanics

How it works

Get probabilities on validation from predict_proba.
Fix the cost matrix: FP and FN costs (try rupees, seconds, alerts).
Expected cost at threshold t = P(FN)*c_fn + P(FP)*c_fp over the validation distribution.
Sweep t in [0,1]; pick the minimum-cost operating point (report the curve).
Re-tune monthly as costs/priors drift; log the chosen threshold in the model card.

Cost curve vs threshold.

Sweep t0..1 probabilities

CostFN*c_fn + FP*c_fp

Minimizeargmin operating point

Deploylog threshold + monitor drift

Hands-on

Examples

Cost-driven threshold selection

import numpy as np
 
rng = np.random.RandomState(0)
y = np.array([0]*970 + [1]*30, dtype=float)
p = np.clip(y + rng.normal(0, 0.4, 1000), 0.01, 0.99)
 
cost_fn, cost_fp = 100.0, 2.0   # miss: $100; false alert: $2
ts = np.linspace(0.05, 0.95, 19)
best = None
for t in ts:
    pred = (p > t).astype(int)
    fn = ((pred == 0) & (y == 1)).mean()
    fp = ((pred == 1) & (y == 0)).mean()
    cost = fn * cost_fn + fp * cost_fp
    if best is None or cost < best[1]:
        best = (t, cost)
    print(f"t={t:.2f} fn={fn:.3f} fp={fp:.3f} cost={cost:.2f}")
print("best operating threshold:", round(best[0], 2), "cost", round(best[1], 2))

With false-negative cost 50x the false-positive cost, the minimum-cost threshold sits well below 0.5 — mathematically 'call more often' wins until alert noise eats the gains.

Real world

In production

Fraud alerts: threshold set by alert-to-review cost; ops reviews even unlabeled queues daily.
Marketing: recall of churners @ 5% contact budget — tune top-k, not a fixed cutoff.
Quality gates: false-dismiss of a bad batch > false-pass, so threshold tilts toward recall.

Guidelines

Best practices

Lock the cost numbers with the business before modeling; recompute the threshold on fresh validation, not the test set; log the threshold to the model card so a month-later reviewer can re-derive it; when costs drift (promo/seasonality), re-tune.

Watch out

Common mistakes

Treating 0.5 as sacred; setting the threshold by eyeballing a curve with no cost; tuning threshold on test; forgetting that prior class ratios shift the optimum operating point.

When things break

Debugging tips

Sharp cost cliff near t=0.1 means most mass is in 'for sure negative' — good; spike near a value means probabilities cluster there (investigate).
Recompute with the current class ratio: pr rate drift moves the optimum.
Cost curve should be U-shaped; if it's monotone, one error type's cost is mis-set.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

Higher FN cost should push the threshold:

Retuning the threshold requires:

Go deeper

Official documentation

Precision-recall curves (sklearn)

docs

Receiver operating characteristic | Wikipedia