Metrics, Logs and Token Counts
The numbers and the raw events
LLM metrics aggregate latency, token spend, cache hits and error rates; logs capture raw events with the trace id. Token counters deserve first-class status — they drive cost dashboards, drift flags and capacity planning.
What you will learn
- Define the LLM metric families and units
- Log events with trace correlation
- Track token spend per dimension
What is this?
Metrics are aggregates (p95 latency, tokens/min, error %, cache hit %), logs are per-event records, and token counts are the currency of LLM cost. All three must carry the same dimensions — model, provider, tenant, endpoint, version — to be sliceable.
Why do you need it?
- Percentiles hide sheep in wolf's clothing; token metrics reveal cost drift early.
- Logs without trace ids are un-joinable to the metric that spiked.
- Token spend is a top KPI: it moves with traffic, price and model choice.
Mental model
Metrics + logs
A gym's records
The dashboard (metrics) shows weekly attendance and injuries; the incident book (logs) holds each exact event. Only together can you ask 'why did Thursday spike'.
How it works
- Counters: requests, prompt tokens, completion tokens, cache reads, cost cents.
- Histograms: TTFT, total latency, tokens/sec, per-model.
- Logs: request start/end, errors, retries, guardrail flags — each with trace_id.
- Token usage usually arrives asynchronously (usage fields on responses); tally it per response.
Metric pipeline.
In production
- Cost dashboard: tokens/min × price, split by tenant and model, with anomaly alerts.
- Capacity: p95 tokens/sec vs GPU pool headroom.
- Quality proxy: log judge verdict distribution as a metric fed from the log stream.
Best practices
- Label everything with model, version, tenant, endpoint.
- Tally token counts from responses, not guesses; include cache read/write splits.
- Keep log retention shorter for raw text, longer for hashed/redacted summaries.
- Alert on cost-per-thousand and token-rate anomalies, not just errors.
Common mistakes
- Asterisk-only metrics with no dimensions — 'model spend' impossible to split.
- Logging token estimates (length guesses) instead of the usage field.
- Storing raw full prompts without redaction and retention limits.
Debugging tips
- Spend spike and traffic flat → model or prompt change; compare per-model tokens/min.
- p95 flat but p99 spiky → a rare long-context path; filter logs by trace length.
- Cache hit rate falls → keys changed (version bump); token read/write counters confirm.
Interview checkpoint
Hands-on lab
- Count prompt and completion tokens from responses
- Group spend by model and tenant
- Compute cost per 1K tokens
Tally Token Spend by Dimension
30 min · intermediate
Quick quiz
Which metric best captures LLM cost?
Logs and metrics must both carry the same dimension labels to be compared.