Model Monitoring
Watching the model after the applause dies down
Model monitoring watches how a deployed model behaves in production: inputs, predictions and their downstream outcomes, plus the platform serving them. Its job is to catch degradation early — drift, staleness, latency, and quality slippage — before users feel it.
What you will learn
- Explain what production model monitoring watches and why
- Separate input, output and outcome monitoring
- Define the life of a prediction event in telemetry
What is this?
Model monitoring continuously observes a deployed model in production: the data arriving at it (inputs), the predictions it makes (outputs), the outcomes those predictions lead to, and the latency/health of serving. It converts 'is the model still good?' from a guess into measured signals.
Why do you need it?
- The world changes: data drift and concept drift degrade models silently, not on schedule.
- Platform metrics alone can't tell you the model is answering worse.
- Early detection is the difference between a quiet retrain and a burned cohort.
- Regulation and auditing increasingly demand evidence of production behavior.
Mental model
A lifeguard watching swimmers, not just the pool's temperature
Model monitoring
Filter cleanliness and chlorine (platform) matter, but the lifeguard watches the actual swimmers (predictions and outcomes). Club rules say log each rescue; a guard who only reads the water diary misses the drowning.
How it works
- Serving stamps each prediction with model version, input snapshot, prediction and timing.
- A logging/sample path persists prediction events to a telemetry store.
- Ingestion computes monitoring metrics: drift, distributions, latency percentiles, quality proxies.
- Thresholds push alerts and feed dashboards and promotion gates.
- Outcome feedback loops reconcile predictions with later-arriving labels.
The monitoring data path from prediction to alarm.
In production
- A fraud model pages when input distributions shift beyond a PSI threshold.
- Latency-by-version telemetry caught a serving regression before users complained.
- Outcome reconciliation flags a churn model whose predictions stopped predicting.
Best practices
- Monitor inputs, outputs and outcomes — not just one of the three.
- Log enough for postmortems: model version, input snapshot, decision and timing.
- Alert on business-relevant signals, not dashboard decoration.
- Wire monitoring into gates so production stays accountable to its own telemetry.
Common mistakes
- Monitoring CPU but not predictions — the model could rot invisibly.
- Logging without the input snapshot, so nothing is reconstructable later.
- Alerts with no owner and no action path — they decay into noise.
- Monitoring the model only at launch; three months later it's watching nothing.
Debugging tips
- Drift alert 'out of nowhere' — check data pipeline changes before blaming the world.
- Telemetry gap — verify the sampler and exporter didn't silently stop.
- Quality dip with no drift — suspect outcome/label definition change, not the model.
Interview checkpoint
Quick quiz
Model monitoring's most important habit is…
A prediction event should carry model version and input details for later reconstruction.