Classification
Intermediate 13 min read Classification

Probability Calibration

Make the probability mean what it says

A ranked model and a calibrated one differ: calibration aligns predicted probabilities with observed frequencies, and is crucial wherever probabilities drive expected-value decisions.

calibrationPlatt scalingisotonic regressioncalibration curvereliabilityprobability
Prerequisites: Logistic Regression
Before we start

What you will learn

  • Read a calibration (reliability) curve
  • Apply Platt scaling and isotonic calibration correctly
  • Never calibrate on data used for fitting the model
Introduction

What is this?

Calibration fixes the gap between predicted probability and observed frequency. A model saying '70%' is calibrated if, across rows predicted at ~70%, about 70% actually happen. Tools: calibration curve (binned predicted prob vs actual rate), Platt scaling (logistic fit on scores), isotonic regression (monotone fit, flexible). Apply on a held-out split; then re-verify.

Motivation

Why do you need it?

  • Ranking-only uses (AUC, top-k) don't need the number to mean frequency; decisions DO.
  • Expected-value optimizations (contact only if p(churn) x LTV > cost) hinge on absolute probabilities.
  • GBMs and SVMs are notorious for distorted probabilities; calibration restores trust.
  • A miscalibrated probability silently bloats or starves alert queues.
Think about it

Mental model

Imagine

A thermometer that ranks days but reads 30° when it's really 20°

Think of

Miscalibration

For 'hotter/colder' ranking it's fine; for choosing a coat (a decision) it's wrong by a lot. Calibration fixes the scale so the number means what it claims.

Mechanics

How it works

  • Get predictions on a calibration-holdout — NOT the training data.
  • Bin rows by predicted probability; compare bin-mean prediction to bin-mean truth -> calibration curve.
  • Platt: fit logistic on the model's scores to re-map S-curves; good for low data.
  • Isotonic: monotone non-parametric fit; strong on plenty of data, riskier on little.
  • Re-validate ONCE on untouched data after calibration; log both curves to the model card.

Reliability curve shapes.

Diagonal
S-curve
Flat/aggressive
Fix
Hands-on

Examples

Sigmoid distortion and isotonic repair

Sigmoid distortion and isotonic repair
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.calibration import CalibratedClassifierCV
 
X, y = make_classification(n_samples=3000, n_features=8, weights=[0.85, 0.15],
                           random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
 
base = RandomForestClassifier(n_estimators=200, random_state=0).fit(Xtr, ytr)
cal = CalibratedClassifierCV(base, method="isotonic", cv=3).fit(Xtr, ytr)
 
raw = base.predict_proba(Xte)[:, 1]
adj = cal.predict_proba(Xte)[:, 1]
 
def ce(p, y):
    idx = np.linspace(0, 1, 11)
    bins = np.digitize(p, idx) - 1
    return np.mean([abs(p[bins == b].mean() - y[bins == b].mean()) for b in range(10) if (bins == b).sum() > 20])
 
print("calibration error raw:", round(ce(raw, yte), 3))
print("calibration error isotonic:", round(ce(adj, yte), 3))

The forest's raw probabilities bow away from the diagonal; the isotonic pass herds the calibration error down while leaving rank order essentially intact.

Real world

In production

  • Expected-value retention: p(churn) drives spend decisions — absolute value must be honest.
  • Alert prioritization: rank is enough, but queue volume math (alert value × p) needs calibration.
  • Regulated default models: regulators inspect reliability curves, not just AUC.
Guidelines

Best practices

  • Calibrate on a dedicated holdout (or within CV as CalibratedClassifierCV does); never calibrate on the same rows that trained the model; verify on untouched data; re-run calibration after any retrain or data drift; document the method (isotonic vs Platt) in the model card.
Watch out

Common mistakes

  • Calibrating on training rows (double-counts); using isotonic on tiny samples (overfit the calibration curve); reporting AUC only and assuming probabilities are trustworthy; calibrating AFTER choosing the threshold (order matters: calibrate, then threshold).
When things break

Debugging tips

  • Reliability curve above diagonal = underconfident (too-low probability for realized rate); below = overconfident.
  • After calibration, the curve should hug the diagonal within noise; re-bin carefully on small data.
  • If calibration 'fixes' the curve but the binned noise is huge, your holdout is too small to trust.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A perfectly calibrated model predicts 0.7; over many 0.7 rows, about ____ are positive.

2

Calibration must be fit on:

Go deeper

Official documentation

Probability calibration (sklearn)
docs
Calibration curve walkthrough (sklearn example)
video
Probabilistic classification | Wikipedia
docs