Monitoring for LLMs
Open-ended output needs new measuring sticks
LLM monitoring is hard because outputs are open-ended: track safety/guardrail hits, cost and tokens, latency, hallucination proxies, and quality via evals and human feedback — not a single accuracy number. The stack adds judges, evalsets and feedback channels to classic telemetry.
What you will learn
- Track guardrail, cost, latency and quality signals for LLMs
- Use evalsets and judges for quality telemetry
- Fold human feedback into the monitoring loop
What is this?
LLM monitoring extends classic telemetry to open-ended text: safety guardrail hit rates, token cost per request, latency/streaming metrics, hallucination and quality proxies via evals + judges, and human feedback loops. There's no single AUC — quality is a basket of signals.
Why do you need it?
- Open-ended output can't be scored against one label; evals and judges proxy quality.
- Guardrail hits and refusal rates are safety-relevant production signals.
- Token cost varies per request and model — a first-class monitoring dimension.
- Feedback (thumbs, corrections, downstream outcomes) is the ground truth humans provide.
Mental model
A call center grading open-ended calls with rubrics, not multiple choice
LLM quality monitoring
Supervisors grade calls on several rubric axes (accuracy, tone, resolution) plus listener feedback (human feedback) and chargeback averages (cost/latency). No single bubble sheet score; the composite decides coaching.
How it works
- Log input/output pairs, model version, latency, token counts and guardrail events per request.
- A judge (LLM-as-judge or components) evaluates sampled outputs against rubric dimensions.
- Feedback signals (thumbs, edits, downstream outcomes) reconcile like labels.
- Cost and latency per request are aggregated by model/version.
- Drift-style signals (topic shifts, refusal-rate changes) trend alongside quality.
LLM monitoring signal flows.
In production
- A support assistant tracks refusal rate, cost per ticket and a judge-scored helpfulness index.
- A content tool alerts when hallucination-suspect outputs spike for a prompt family.
- A chatbot monitors token-cost creep per model version to catch prompt regressions.
Best practices
- Log every request pair with version and guardrail events; sample for judge eval.
- Maintain a live evalset that mirrors real traffic, refreshed from feedback.
- Use varied judges plus human spot-checks to contain judge bias.
- Monitor cost and latency as first-class quality signals — they're user-facing too.
Common mistakes
- No judge, no evalset — quality is 'asked three people once', then never again.
- Trusting the LLM's own judge scores exclusively (self-bias).
- No guardrail telemetry, so a safety regression surfaces only in incident reports.
- Ignoring token cost until the invoice — cost belongs in daily monitoring.
Debugging tips
- Quality 'improved' but users unhappy — judge rubric drifted from user behavior; refresh evalset from traffic.
- Cost spike not in eval — measure tokens on realistic requests, not the tiny eval fixture.
- 'Silent' hallucination rise — a judge-eval on sampled outputs is what catches open-ended decay.
Interview checkpoint
Hands-on lab
- Log requests with version, tokens and guardrail events
- Run a simple judge over a sample evalset
- Aggregate cost and refusal-rate metrics
Wire Basic LLM Monitoring Signals
35 min · advanced
Quick quiz
LLM quality monitoring replaces a single accuracy metric with…
Token cost and latency are legitimate LLM monitoring dimensions.