Probability Calibration

• Diagnose miscalibration with reliability curves and Brier score
• Apply Platt and isotonic rescaling and know their trade-offs
• Explain why neural-net softmax outputs are not calibrated probabilities

• Decision rules like 'take action when risk > 30%' are only correct if 0.3 means 30%.
• Insurance, lending and medicine price risk off probabilities, not just rankings.
• Neural networks overconfident by default: softmax hoodwinks even experienced teams.
• A model can win on AUC and lose money on calibration.

'70% chance of rain' is only useful if, on the days you say 70%, it rains seven times out of ten. The forecaster is graded on calibration, not on whether any single forecast was right.

• Collect the model's scores on a held-out set along with the true labels.
• Bin the scores (e.g., deciles); per bin, compare the mean predicted score to the observed positive rate.
• Plot predicted vs observed — points on the diagonal are perfectly calibrated; below the line means over-confidence.
• Quantify with Brier score = mean((p_hat - y)²), lower is better; decompose into calibration + refinement.
• Remedy: fit a monotonic transform on a separate set — Platt logistic regression for smooth curves, isotonic regression for data-rich, flexible fits.

• A credit-risk engine rescales scores with isotonic regression before feeding pricing, because a '0.9 default risk' must literally mean a 90% rate.
• An ICU mortality model gates ICU admission on calibrated risk thresholds — miscalibration would mis-triage patients.
• A churn team uses reliable probabilities to rank saving-campaign spend by expected lift, not just rank order.

• Always fit the calibration transform on data the model didn't see (a calibration hold-out or inner fold).
• Diagnose with a reliability curve before rescaling — the shape tells you which fix fits.
• Retrain calibration regularly: drift makes previously-calibrated scores unreliable.
• Report Brier score, which is threshold-independent, in addition to any threshold metrics.

• Reading softmax/sigmoid outputs as calibrated just because they lie in [0,1].
• Fitting Platt/isotonic on the training set, double-dipping the data and overfitting the remap.
• Assuming calibration is permanent — deployed distributions shift, so re-verify periodically.
• Applying calibration to change the ranking; monotonic rescaling preserves order but distorts separation.

• Reliability curve S-shaped below the diagonal? Overconfidence — common for boosted trees on separable data.
• Brier better on train than test? The calibration fit is overfit; use a larger calibration set or simpler Platt.
• AUC unchanged after calibration is expected — calibration improves probabilities, not ranking.
• If the curve is all over the place, the bins are too small; use quantile bins.