Observability on K8s
Platform health vs model health, both visible
The cluster generates the three signals — metrics, logs, traces — and ML adds model-level telemetry on top. Observing K8s means wiring Prometheus-style metrics, structured logs and traces while distinguishing platform health from model health.
What you will learn
- Collect metrics, logs and traces for ML workloads on K8s
- Separate platform signals from model-decision signals
- Build dashboards/alerts that cover both layers
What is this?
Observability on K8s layers model telemetry over platform telemetry: cluster metrics (Prometheus), structured logs (Loki/EFK), and traces of request paths, plus model-level metrics — predictions, drift, latency by version. The trick is knowing which layer a signal reports on.
Why do you need it?
- A 'slow model' is often a mis-labeled platform problem (node pressure, throttling).
- Model changes are invisible in CPU/memory alone — you need prediction-level metrics.
- Alerts that conflate the two send the wrong team chasing the wrong signal.
- Traces connect a request's path across serving, features and cluster components.
Mental model
A hospital's vitals plus lab results
Platform + model observability
Vital signs (CPU, memory, restarts) say the machinery runs; lab results (prediction metrics, drift) say the treatment works. Doctors act differently on a fever you found than a blood test — because they measure different truths.
How it works
- A metrics pipeline (Prometheus + exporters) scrapes kubelets, kube-state-metrics and app endpoints.
- Structured logs with correlation IDs flow to a centralized store.
- Traces capture request hops: ingress → Service → model pod → feature store.
- Model telemetry appends prediction-level counters/latencies/version labels.
- Dashboards and alert rules separate platform SLOs from model-quality SLOs.
Telemetry layers on K8s ML.
In production
- A node-pressure alert vs a prediction-diff alert page different teams.
- Traces show a fraud request's feature-store hop costing 40ms — model is fine.
- Dashboards split 'cluster utilization' from 'model p95' with cross-linked drill-downs.
Best practices
- Emit model endpoints at HTTP-level metrics (prometheus_client) labeled by version.
- Correlate logs/traces with a request id spanning serving and features.
- Alert separately: platform SLOs vs model-quality SLOs, with distinct ownership.
- Cap metric cardinality — version labels yes, per-request refs no.
Common mistakes
- Only platform metrics — a behavioral regression looks like 'nothing happened'.
- Model metrics unlabeled by version — 'better or worse' is undecidable after rollout.
- Verbose unstructured logs nobody joins to requests.
- Alert rules that page on both platform noise and real model drift.
Debugging tips
- Slow responses — first split symptom by layer using traces before touching the model.
- Prediction drift alert — confirm it's model quality, not a data pipeline change.
- Metrics gap — exporter scrape targets misconfigured; verify target discovery.
Interview checkpoint
Hands-on lab
- Emulate Prometheus-style counter/histogram metrics
- Label all model metrics by model version
- Build a per-layer alert heuristic
Emit Version-Labeled Model Metrics
35 min · advanced
Quick quiz
To tell if a new model version is genuinely faster, you need…
Platform SLOs and model-quality SLOs should be alerted separately.