Monitoring Pipelines
Automating the analysis that keeps models honest
Model analysis runs on pipelines, not ad-hoc queries: scheduled jobs that compute drift, reconcile outcomes, and emit quality metrics back into the telemetry store. Automating the analysis is what keeps monitoring alive without a human running SQL weekly.
What you will learn
- Design scheduled jobs for drift, reconciliation and quality
- Write monitoring results back into the telemetry store
- Trigger retraining or alerts from pipeline findings
What is this?
Monitoring pipelines are scheduled data jobs that do the model analysis automatically: compute drift vs baseline, reconcile predictions with arriving labels, emit quality metrics to the store, and trigger alerts or retraining. They keep monitoring alive without a weekly human SQL session.
Why do you need it?
- Manual analysis decays — someone stops running it and monitoring dies.
- Scheduled jobs make drift and quality trends continuous and comparable.
- Pipeline findings can act: fire alerts, open tickets, trigger retraining.
- Results in the telemetry store surface on the same dashboards as everything else.
Mental model
Streetlights on a timer plus a fault-reporting circuit
Monitoring pipelines
The lights (checks) switch on automatically every night (schedule); the circuit (pipeline) reports burned bulbs (alerts) and the city's response is policy, not hoping a passerby files a complaint.
How it works
- A scheduler (Airflow/cron) runs the monitoring DAG on a cadence.
- The drift job reads a sampled window and the baseline, computes statistics.
- The reconciliation job joins predictions with labels that arrived in the window.
- Metrics are written back to the telemetry store with model/version labels.
- Thresholds translate findings to alerts, tickets or retraining triggers.
Scheduled monitoring pipeline outputs.
In production
- A nightly drift job writes PSI per feature into Prometheus for dashboards.
- A weekly reconciliation job computes AUC on labeled windows and gates promotion.
- A drift breach pipeline triggers a retraining run through the training pipeline.
Best practices
- Treat monitoring jobs like production: versioned, tested, observable themselves.
- Pin the window and baseline versions the job compared.
- Write results with model/version/stage labels for consistency.
- Make the pipeline's own failures loud — an unrun drift job is a silent gap.
Common mistakes
- Bench queries that nobody scheduled — monitoring relied on memory.
- Drift jobs with an unpinned baseline, so 'drift' includes baseline movement.
- Pipeline failure silent; monitoring silently stops, dashboards freeze at 'ok'.
- Analysis thrown away — computed but never emitted, never alerting.
Debugging tips
- 'Drift frozen at old values' — the job stopped running; check its scheduler last-run timestamp.
- Baseline skew — the job may be comparing to a drifting/re-written baseline; pin it.
- Pipeline slow — sampled windows too large; sample down for the metric granularity needed.
Interview checkpoint
Quick quiz
Monitoring pipeline results should…
A monitoring job's own failure should be visible and noisy.