Production ML
Advanced 16 min read Production ML

Monitoring Models in Production

Watch the inputs, the outputs and the smoke signals for quality

Monitoring models means watching the signals that predict decay: input distributions, prediction distributions, latency and proxies for quality — because true labels arrive late. Alerts, dashboards and review cadence make decay visible before users feel it.

monitoringdriftproxy metricground truthprediction distributionalertingSLO
Prerequisites: Production ML Architecture Metrics
Before we start

What you will learn

  • Choose metrics to monitor when labels are delayed or absent
  • Detect drift in inputs, predictions and calibration proactively
  • Design alerting thresholds, dashboards and an SLO review cadence
Introduction

What is this?

Live monitoring watches the observable signs that a model's edge is stale: the distribution of incoming features (data drift), the distribution and confidence of predictions (prediction drift), latency and error rates, and quality proxies where true labels are delayed — e.g., click-through for a recommender, dispute rate for fraud, or human-review adoption rate. Alerts compare rolling windows against baselines, and a review dashboard gives operators a standing view of model health.

Motivation

Why do you need it?

  • True labels (fraud confirmed, purchase completed) arrive minutes to weeks late — you must monitor anyway.
  • Silent decay is the most expensive failure mode: users experience degradation while dashboards look green.
  • Predictive monitoring (drift) gives hours-to-days of lead time before quality visibly drops.
  • An SLO with review cadence makes 'is this model still fine?' answerable by anyone, any day.
Think about it

Mental model

Imagine

Vital signs, not just the annual checkup

Think of

Model monitoring

You don't wait for the annual physical to discover heart trouble — you watch resting heart rate, weight and blood pressure continuously and act on trends. Model monitoring is the vital-signs layer; the annual checkup is the label-based evaluation that confirms what trends already warned about.

Mechanics

How it works

  • Log every prediction event: features, prediction, decision, metadata (version, request context).
  • Compute rolling statistics: feature distributions, prediction mean/quantiles, calibration, missing rates.
  • Compare to baselines: static (training distributions) and dynamic (sliding windows) with statistical tests.
  • Compute proxy quality: for delayed labels, correlate predictions with the closest behavioral outcome.
  • Alert on thresholds with hysteresis; review on a cadence in a dashboard that operators actually read.

What monitoring watches, and what it triggers.

Log predictions
Compute stats
Compare baselines
Proxies
Alert & review
Real world

In production

  • A fraud team monitors prediction rate and dispute rate per score band; a spike in high-score-but-no-dispute suggests the model is over-flagging.
  • A recommender tracks click-through and coverage per cohort; drift in cohort coverage triggers a review before ranking quality collapses.
  • A healthcare model monitors feature missing rates and calibration in production traffic, with a weekly human review of low-confidence cohorts.
Guidelines

Best practices

  • Monitor inputs, outputs AND at least one outcome-linked proxy — never only one.
  • Set alert thresholds with hysteresis and noise filtering; pager fatigue kills the monitor.
  • Snapshot inputs with every prediction so 'wrong because of X' is answerable post-hoc.
  • Define a golden review (weekly) combining drift, latency and proxy trends into an owner decision.
  • Include the model version in every log line — monitoring without versions is archaeology.
Watch out

Common mistakes

  • Monitoring infrastructure (CPU/latency) and calling it done — model behavior is the actual risk.
  • Alerting on single-request deviations, flooding pages with noise and training everyone to ignore them.
  • Ignoring delayed labels entirely; the proxy metric never gets reconciled against ground truth.
  • Drift alerts without an owner or runbook — 'we know' means 'we don't do anything'.
Root cause analysis

Common issues

Error
Dashboard is green but users complain quality dropped
Root cause

Monitoring misses outcome-linked proxies and only watches latency/uptime

Fix

Add behavioral proxies and drift sensors tied to the decision being made.

Error
Drift alerts fire constantly
Root cause

Threshold too tight or a stale baseline (training data no longer matches reality)

Fix

Rebaseline the drift comparator and add hysteresis plus aggregate windows.

Error
Alerts ignored by operators
Root cause

No runbook or owner attached to the alert

Fix

Attach ownership, runbooks and expected-action steps to every alert before rollout.

When things break

Debugging tips

  • When a proxy dips, check whether inputs drifted, decisions shifted, or the code updated — three different worlds.
  • Compare the prediction distribution to training; a collapsed one means the model is defaulting on edge inputs.
  • Reconcile proxy vs true labels on a lagged cohort to calibrate how trustworthy your smoke signals are.
  • Time-box investigation to the logged feature snapshot; without it, every hypothesis is speculation.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Live Drift + Proxy Monitor
35 min · advanced
In progress
Objectives
  • Simulate a live prediction stream with inputs that drift over time
  • Compute a rolling drift score versus the training baseline
  • Alert on drift and demonstrate a proxy-quality signal decaying in lockstep

Build a Live Drift + Proxy Monitor

35 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Labels arrive 14 days late. Which signal should you monitor NOW?

2

A model's prediction distribution changing means the model is broken.

Go deeper

Official documentation

Amazon SageMaker Model Monitor
docs
MLOps: continuous delivery pipelines
docs