Model Performance Monitoring
Is the model still good, and how do you know in live time?
Performance monitoring watches whether the model is still good, not just healthy. Offline evaluation measures a fixed candidate; online evaluation measures the live truth — with delay, bias, and noise the price of reality.
What you will learn
- Distinguish offline vs online performance evaluation
- Account for label lag and selection bias in live metrics
- Choose the right metric per business outcome
What is this?
Performance monitoring is the practice of measuring model quality continuously. Offline evaluation runs against fixed historical splits; online evaluation measures real-world performance with labels or proxies — introducing label lag, selection bias, and noise you must design around.
Why do you need it?
- A healthy (non-crashing) model can silently be wrong where it matters.
- Live data is what reality is; offline splits age into fiction.
- Label lag and bias force you to design honest online metrics.
- Separating offline strictness from online honesty locates where losses come from.
Mental model
A car's dyno test vs real highway driving
Offline vs online evaluation
On the dyno (offline), controlled and repeatable, but nobody drives on a dyno. On the highway (online), traffic, weather, and other drivers — replicate reality but noisier and delayed by the road. You need both numbers, and the gap between them is the interesting part.
How it works
- Offline: fixed split, unbiased labels, repeatable — but aging realism.
- Online: live predictions matched to eventual outcomes.
- Handling the label-age problem — error grows honest over time but lags now.
- Bias control — compare like-for-like (per segment, per time) for unbiased online rates.
- Combine: offline picks candidates, online verifies reality, gap-reconciled continuously.
Offline + online loop.
In production
- A Credit model runs offline ROC gates pre-deploy, then watches monthly realized default rate online.
- A Churn model's offline AUC looked great; online retention-by-cohort shows the decay.
- A Recommender mixes offline ranking quality with online CTR/engagement as the live readout.
Best practices
- Maintain a champion-challenger pipeline so online comparison is always computable.
- Standardize on label-waiting windows so metrics compare fairly.
- Segment online metrics — reality lives in the slices.
- Track the offline-versus-online gap; divergence is the first warning bell.
Common mistakes
- Comparing new offline metrics against old online metrics on different populations.
- Slamming the model for online 'badness' measured before labels matured.
- One global number hiding the segment where the model is actually dying.
- Designing the 'perfect' offline harness while live instrumentation rots.
Debugging tips
- Online < offline persistently — check population shift or a stale label pipeline, not just the model.
- Online metric jumps every month that ARCOS-like pipeline metrics don't — question the label window.
- Candidate beats champion offline but loses online — the offline split didn't resemble the live drift.
Interview checkpoint
Quick quiz
Offline evaluation's chief aging flaw is…
Online evaluation is always real-time and unbiased.