Cost Monitoring and Budgets
See the bill while it's being written
Spend lives per tenant, model and endpoint. Instrument token counts at the gateway, budget by dimension, and alert on spend anomalies plus cost-per-request regressions — catch overspend before the invoice, not after.
What you will learn
- Instrument token spend per dimension
- Set budgets by tenant or feature
- Alert on spend anomalies and per-request drift
What is this?
Cost monitoring aggregates token usage and price per request, bucketed by tenant, model, endpoint and version. Budgets bound each dimension; alerts fire on anomalous spend growth and cost-per-request regressions — the signal of prompt bloat or routing drift before the invoice lands.
Why do you need it?
- Bills arrive weeks after the overrun; gateway tagging is real-time.
- Per-tenant budgets stop one noisy customer from blowing the shared bill.
- Cost-per-request drift reveals regressions (bigger prompts, higher tier) invisible in totals.
Mental model
Monitoring
A taxi meter plus a weekly budget
You watch the meter per ride (per request) and reconcile against a weekly cap (budget) — you never learn the fare at month end.
How it works
- Gateway/retriever emits tokens + price per call with tenant/model/version tags.
- Aggregate spend per dimension into dashboards.
- Set budgets (tenant, feature, monthly total) with burn-rate alerts.
- Alert on cost-per-request percentile drift, not just totals.
Spend loop.
In production
- A SaaS slices cost by plan tier and alerts when a grandfathered tenant grows 3x in spend.
- A platform tracks cost-per-request p95 and gets paged when a model swap bumps it.
- A finance team reconciles gateway-estimated spend vs the provider invoice monthly (±1%).
Best practices
- Tag every call with tenant, model, endpoint, version at write time.
- Alert on percent change with a lookback — seasonal traffic holds steady, blips don't.
- Reconcile estimates to invoices; currency/price-table drift shows up early.
- Make extrapolated month-end spend a visible number in every review.
Common mistakes
- Only total spend visible — no dimension to drill into when it grows.
- Static alerts: threshold spends forever, seasonal burst pages at midnight.
- Ignoring invoice reconciliation, so price-table update surprises everyone.
Debugging tips
- Spend up, traffic flat → average prompt tokens or tier mix changed; diff by version.
- One tenant ballooning → their traffic pattern changed; check their cache-hit rate and prompt size.
- Cost-per-request spike → routing sent more to frontier; inspect model distribution per request.
Interview checkpoint
Quick quiz
The earliest signal of a routing regression to cost?
Spend should be sliced by tenant, model and endpoint.