ML Cost Optimization
Every epoch needs an invoice
ML cost management tracks spend across training, serving, storage, and data — mapped to the value each model creates. Optimize by matching instance shape to workload, right-sizing, budgeting, and making cost a first-class release metric.
What you will learn
- Break ML spend into training, serving, storage, data buckets
- Attach cost per model to business value
- Set budgets, alerts, and optimization loops
What is this?
ML cost optimization is continuous: instrument spend by workload, model, and stage; right-size instance and storage; exploit spot and autoscaling; and govern with budgets and unit-economics so 'it runs' is never confused with 'it earns'.
Why do you need it?
- ML spend scales with GPU-density — it's a new league of overspend.
- Unattached compute and stale clusters burn money silently.
- Unit economics (cost per decision/prediction) is what leadership actually feels.
- Visibility plus governance is the only thing stopping unbounded growth.
Mental model
A kitchen where each dish gets a cost card
ML cost tracking
Chefs (teams) cook all winter; the owner (you) adds up one huge bill without a per-dish card. You can't cut, optimize, or justify any 'menu' (model) unless each has ingredients, labor, and margin tracked. Cost cards make the business of cooking legible.
How it works
- Instrument spend per stage: training (GPU-hours), serving (nodes/model), storage (datasets/checkpoints), data (ingest/transform).
- Attach cost to unit economics: the model's cost per decision, prediction, or outcome.
- Right-size: match instance shapes and counts to workload profile (latency vs throughput).
- Exploit elasticity: spot for interrupts, autoscale serving to traffic, idle-scaling training.
- Govern with budgets, alerts at thresholds, and monthly cost-to-value reviews.
ML cost loop.
In production
- A serving team discovers a stale 96-GPU cluster orphaned by an old training job — deletes it, saves 40% of the month.
- A recommender's cost-per-served-rank drops 3x after autoscaling kicks in at off-peak.
- A monthly cost review kills two experiments whose value (lift) never appeared — freezing real savings.
Best practices
- Track cost per model and stage on dashboards owners sign weekly.
- Set hard spend alerts; budget per tenant/team before the strain.
- Right-size by profiling, then assert on instance choice.
- Make cost a release gate — a no-value model is a cost incident.
Common mistakes
- One giant 'ML' line item with no owner breakdown.
- Buying the biggest GPU 'just in case' for a latency-tolerant batch job.
- Forgetting storage: checkpoints and data replicas outspend compute over time.
- Zero governance: budgets exist on paper only.
Debugging tips
- Spike in GPU hours beyond jobs — look for orphaned pods, retries, and daemon jobs.
- Cost up, value flat — re-derive unit economics; the model quietly stopped earning.
- Storage bill dominating — prune checkpoints and pointer data you no longer read.
Interview checkpoint
Quick quiz
The most silent ML cost leak is usually…
A model that costs more but earns more per unit is a cost incident.