ML Cost Budgeting and Anomaly Detection

• Set budgets per team, model, and environment
• Detect cost anomalies beyond hard thresholds
• Turn cost events into governed actions, not surprises

• ML spend has long tails: one runaway retrain can spend a quarter's allocation in a day.
• Hard dollar caps miss moderate but sustained drift toward a slow bleed.
• Budgets make cost someone's job, not a month-end shock.
• An anomaly is only useful when there's a decided response attached.

If you only react when the yearly budget says 'you're done in October', December's heater plus swimming-pool rumor (runaway job) goes unnoticed. The app compares against the Sunday baseline — this one bill is 3x normal; it flags it now. That's anomaly-with-evidence, before panic.

• Allocate budgets by tenant, model, environment — signed by an owner.
• Forecast spend from trend and burn-rate; run a budget burn-rate projection.
• Anomaly detection on spend series (z-score vs rolling baseline, seasonal split).
• Alert on: budget threshold, burn-rate forecast breach, anomalous spikes and persistent drift.
• Governed actions: throttle, tag-and-park, freeze unapproved runs, notify owner with evidence.

• A weekly forecast shows the retraining path breaching Q3 GPU budget by August — the team trims experiment frequency in advance.
• An anomaly alarm catches a 6x nightly budget spend from an orphaned dead-retry loop, auto-parked before the month's gone.
• A cost dashboard per model ties to cost-per-decision so 'expensive' models get a value review, not just a line item.

• Assign each budget a named owner; review against value monthly.
• Alert on burn-rate-forecast too, not just hard ceilings.
• Use seasonal/rolling baselines so anomaly is meaningful, not calendar noise.
• Pre-decide the action per alert severity: notify → tag/park → throttle.

• Budgets with no action policy — the alarm fires; nobody knows what to do.
• Hard-cap-only alerts missing the slow sustained bleed.
• Anomaly baselines ignoring weekends/holidays — weekend 'spikes' become noise.
• Chasing the anomaly instead of the owner — every alarm should name its tenant and plan.

• Anomaly fires weekly at the same hour — that's seasonal, not anomalous; build the seasonal baseline.
• Burn-rate forecast says breach in 3 weeks — start the review now, not at the ceiling.
• Throttle killed a legit job — the action policy was too blunt; grade by tenant criticality.