Cost Monitoring & Anomaly Alerts
Catch spikes before the invoice, not after
Watch the meter live: cost dashboards, budget alerts (Saas, nested), anomaly detection, and run-rate forecasting. Alerts turn 'we found it on the invoice' into 'we catch it in days' — with an escape path (kill a runaway) and a post-discovery debrief.
What you will learn
- Build live cost dashboards per workload/team
- Set budget + anomaly alerts with owners
- Have a run-away response plan (kill, notify, debrief)
Cost observability is like regular observability: dashboards (per service/tag), budget alerts (thresholds), anomaly detection (ML on spend patterns), plus run-rate forecast. The catch: alert with an owner and a response path, else the alert is theatre.
What is this?
Dashboards: Cost Explorer views, Azure Cost Management + Power BI, GCP billing dashboards; per-workspace filters (env/team/service). Alerts: budget thresholds (actual + forecast) at 50/80/90/100%, anomaly detection (AWS Cost Anomaly, Azure cost alerts, GCP budget alerts) tuned per monitor (e.g., +$500 in a day). Run rate: current month projected vs budget; forecast vs commitment. Alert piping: Slack/teams + owner; severity by size + blast radius. Response: the 'runaway breaker' — a playbook to stop a runaway (stop ASG/instance scale, pause imports) with approval gate; notification to finance/leadership. Debrief: post-spike retro: what fired, what it cost, the guardrail (budget/anomaly/policy) that prevents recurrence. End-to-end test: kill an instance → alert fires → owner notified (the whole chain rehearsed).
Why do you need it?
- The invoice is a month too late; monitoring is the difference between a $50,000 surprise and a $1,000 catch.
- Anomaly alerts catch 'the box I forgot to stop' that budget totals can't.
- Lead-time is money: a proper alarm chain catches in days, not weeks.
- Run-rate forecast answers 'can we finish the quarter on budget?' (finance's favorite question).
Mental model
A car's fuel gauge vs. the empty-tank on the highway
Cost monitoring
The gauge plus a light plus a reserve plan is cheap insurance; discovering empty at highway speed (the invoice) is an expensive lesson. Monitor the dash before the light, and know what happens when the light comes on.
How it works
- Cost dashboards per service/tag with freshness ≤ daily (or streaming where possible).
- Budget alerts + anomaly monitors with owners + channels.
- Forecast model: run-rate vs budget; adjust the alert baselines.
- Escape hatch: runaway-breaker playbook (approved stop) + notification.
- Debrief loop: every spike → guardrail added, owner named, forecast re-based.
Cost watch path.
Examples
Alert wiring
# Budget: prod-platform $18k/mo# alerts: 50% / 80% / 90% / 100% (actual + forecast) → #finops + owner# Anomaly: any service +$1,000/day vs 14d median → Slack #cost-alerts# Escape: playbook "stop_import_pipeline" (approval: 2-person) — pause the batch# Debrief: each spike → add guardrail/owner, re-base forecast.Four layers: budget, anomaly, escape, debrief — the whole cycle, not just a beeping alarm.
In production
- Budgets per team/env with 50-100% alerts + forecast alerts.
- Cost Anomaly tuned per workload with a 24h lookback; wire to PagerDuty/Slack.
- Runaway-breaker runbook (documented, drilled) for the top cost-leaking job.
Best practices
- Every alert needs an owner + channel + response — untended cost alarms are compliance theater.
- Baseline anomaly monitors on real 14-30d history to reduce noise.
- Test the chain: force a small spike and confirm the alert+notification path.
- Pair budget alerts with run-rate forecast so 'on pace' is visible before month-end.
- Keep the escape hatch documented and rehearsed (a stop that's never drilled is a stop that's dangerous).
Common mistakes
- Budget only actual, not forecast — the 'we'll be fine' bug shows at month-end.
- Anomaly alerts without noise control: 30/day = disabled forever.
- Kill-switch nobody knows about or that isn't rehearsed.
- Responding to spikes monthly instead of a documented debrief+guardrail each time.
Debugging tips
- Alert floods → tighten thresholds/perms; baseline to the 14-day shape.
- Forecast misses consistently → the model ignores seasonality/batch run changes; feed line items + workdays.
- Anomaly fires but nothing changed → stale tag attribution; wallet it as 'keep an eye'.
- Spike passed the budget completely → alert chain lag or a plan-das threshold missed; audit the alert timing.
Interview checkpoint
Quick quiz
The alert that detects 'something weird got spun up' best:
An untended cost alert is equivalent to: