What to Monitor
The short list of signals that actually matter
A production model earns a matrix of signals: data drift, prediction drift, latency, error rates, and quality with and without labels. The art is picking the few that signal real problems and pairing them with the thresholds that matter.
What you will learn
- Lay out the monitoring matrix for a deployed model
- Pick per-model metrics that signal real decay
- Set thresholds from baseline, not vibes
What is this?
'What to monitor' is a deliberate matrix: data drift and missing rates on inputs; prediction distribution, latency and crash rates on outputs; and quality — offline eval where labels exist, proxy/feedback metrics where they don't. Each model picks the subset its failure modes make valuable.
Why do you need it?
- Monitoring everything is noise; monitoring nothing is blindness.
- Different models rot differently: a recommender decays in ranking joy, a fraud model in recall.
- Thresholds from baseline distributions beat arbitrary constants.
- A tight signal list keeps alerts actionable and investigations finite.
Mental model
A physical exam checklist, not a full genome scan
Choosing monitoring signals
The doctor runs the tests the symptoms and history suggest — blood pressure, cholesterol, glucose — each chosen because it predicts real outcomes at acceptable cost. You don't sequence everyone's genome on a routine visit.
How it works
- Define the model's failure modes: what does 'bad' look like and what causes it?
- For inputs: monitor drift (PSI/KS), missing rates, type/range violations.
- For outputs: monitor prediction distribution, null responses, latency p50/p95, error rate.
- For quality: monitor eval-with-labels and business feedback where possible.
- Set thresholds from a baseline window; escalate when trends confirm, not single spikes.
The monitoring matrix.
In production
- A credit model watches missing rate and KS drift weekly; a spike preceded a real decay.
- A recommender tracks prediction distribution and click-through as its no-label quality proxy.
- A speech model monitors latency p95 and per-version silence rates.
Best practices
- Start from failure modes; add signals a model's decay actually manifests as.
- Pair every alert with a threshold from a baseline window and an owner.
- Sample heavily for monitoring; follow the 1-10-100 cost rule per prediction class.
- Review the monitoring matrix whenever the model or data changes materially.
Common mistakes
- Ten dashboards, zero thresholds — everything visible, nothing actionable.
- Alerting on a single spike; drift trends need windows to be trustable.
- Only output metrics — again, inputs move first.
- Thresholds guessed at 0.05 out of nowhere while the baseline was 0.5.
Debugging tips
- Alert fires constantly — retune using a representative baseline window, not production extremes.
- 'Nothing monitored' for a new model — the matrix is a per-model deliverable, add it at deploy.
- Obscure dip — drill the metric by slice (segment, time-of-day) before changing anything.
Interview checkpoint
Hands-on lab
- Emit inputs/output/quality metrics for a fake model
- Derive a threshold from a baseline window
- Raise a confirmed-trend alert
Build a Monitoring Metric Set
35 min · intermediate
Quick quiz
Which signal usually moves FIRST when a model begins to rot?
Thresholds should be derived from baselines rather than arbitrary constants.