Monitoring and Observability Stack
Turning prediction events into dashboards and alarms
The stack turns prediction events into visibility: emit telemetry, ship it to a store (Prometheus for metrics, a log/index store for events, tracing for request paths), and surface it on dashboards and alerts. MLOps adds model-ready pipelines on top: drift jobs, samplers, and feedback sinks.
What you will learn
- Map data path: emit → ship → store → query → alert
- Choose stores for metrics, logs and traces
- Add ML-specific pipelines (samplers, drift jobs, feedback sinks)
What is this?
The monitoring stack collects prediction events and serving telemetry, ships them to the right stores — Prometheus-style for numeric metrics, log/index stores for events, tracing for request paths — and turns them into dashboards and alerts. MLOps layers model-specific jobs (drift, samplers, feedback) on top.
Why do you need it?
- A signal only becomes visibility once it's stored, queried and shown.
- Different data shapes need different stores: counters vs events vs traces.
- Dashboards make trends legible; alerts make them actionable.
- ML pipelines (drift, feedback reconciliation) automate the model-specific analysis.
Mental model
Air-traffic control: radar screens, flight strips, and the banks of alarms
The observability stack
Radar (metrics) shows the sky; flight strips (logs) narrate each flight's story; the alarm bank (alerting) gets attention when something matters. Remove any wall and the tower can't run the sky.
How it works
- Apps/emitters expose Prometheus-formatted metrics; loggers write structured events; tracing propagates request ids.
- Scraper/pushers ship metrics to Prometheus/Mimir; logs to Loki/Elastic; traces to Jaeger/Tempo.
- Queries and dashboards aggregate into views; alert rules evaluate thresholds.
- ML pipelines compute domain metrics: drift jobs on sampled windows, feedback reconciliation, per-version latency.
- Retention and sampling policies keep the store health and costs sane.
The observability data path.
In production
- Prometheus scrapes model endpoints; Grafana dashboards split platform vs model SLOs.
- A sampler logs every 10th prediction with a full input snapshot to a log store.
- A nightly drift job reads sampled windows and writes drift metrics back into Prometheus.
Best practices
- Route data by shape: counters→Prometheus, events→logs, paths→traces.
- Sample intelligently; log full details for the interesting tail.
- Label metrics with stable dimensions (model, version, stage) and cap cardinality.
- Automate model-specific analysis as scheduled pipelines, not manual queries.
Common mistakes
- Dumping everything into one store and one dashboard graph.
- No sampling — ingestion cost explodes with prediction volume.
- Labels with unbounded cardinality (request IDs in metrics).
- Alert rules written once and never owned, decaying into mute dashboard alerts.
Debugging tips
- 'Dashboard shows nothing' — check scraper targets and the exporter's scraped endpoint first.
- 'Alerts stopped' — verify the rule's query returns data; silence and absence look identical.
- 'Drift panel empty' — the sampling pipeline job may have failed; check its scheduler status.
Interview checkpoint
Hands-on lab
- Emit model metrics in Prometheus exposition format
- Simulate scraping into a queryable store
- Wire an alert rule over the stored series
Pipeline Telemetry to a Queryable Store
40 min · intermediate
Quick quiz
Counters belong in a time-series store because…
Sampling policies are optional when your prediction volume is large.