GPU Cost Drivers
The per-second price of tensor math
GPU spend is driven by model size, batch efficiency, data movement, and cloud pricing shapes. Attack the real levers: utilization, correctness, and instance-shape matches — not just 'buy cheaper GPUs'.
What you will learn
- Decompose GPU cost into time, shape, and utilization
- Right-size ML workloads by profile
- Measure utilization to separate waste from work
What is this?
GPU cost drivers: price of the chosen instance shape, the wall-clock time it's reserved, and the utilization you actually extract — dwarfed by over-provisioning, idle reserve, data-transfer stalls, and epidemic shape-mismatch between job and instance.
Why do you need it?
- A 2-second job renting 8 GPUs costs 8x; utilization is the cheapest 'discount'.
- Shape mismatch (throughput-heavy on latency-optimized) multiplies cost per unit work.
- Data movement (feature load, checkpoint I/O) often under-utilizes the GPU.
- Spot and right-sizing unlock 30-70% without touching model quality.
Mental model
Renting a jet to carry a letter across the street
GPU shape mismatch
The jet (A100) is great for heavy cargo; for a letter (a small fine-tuning job or a tiny latency-capped service) a delivery scooter (right-sized instance) lands the same value at a fraction of the rental. Wrong metal for the job, and the fuel (money) burns twice.
How it works
- Profile utilization: GPU% over time, memory, stall reasons (data load, sync).
- Map workload profile → shape: batch throughput jobs vs streaming latency services.
- Control batch size, accumulation, and parallelism to keep utilization high.
- Choose on-demand vs spot vs capacity discount by interrupt tolerance.
- Right-size iteratively; re-measure after each change, not just after the big one.
GPU cost decomposition.
In production
- A fine-tuning team switches from an on-demand A100 to spot V100 for interrupt-tolerant runs — 3x cheaper per run.
- A serving group moves from 8 instance copies to 2 bigger ones with batch inference and higher utilization.
- A data-loading bottleneck (fetch 4s, compute 1s) fixed by prefetch scheduling — GPU utilization jumps from 40% to 90%.
Best practices
- Instrument GPU utilization and stall reasons before any cost change.
- Right-size by workload profile, not fashion or 'biggest available'.
- Batch greedily on throughput jobs; keep latency lines short and scaled.
- Discount interrupts-tolerant training with spot; nurse critical jobs on reserve or on-demand.
Common mistakes
- Reserving the largest GPU for every job — the 'just in case' tax.
- Tuning batch size into OOM churn rather than utilization.
- Buying cheaper GPUs but shipping the same wall-clock waste.
- No utilization telemetry — can't cut what you can't see.
Debugging tips
- GPU% stuck at 15% while you wait — the stall is data or sync, fix the pipeline not the instance.
- Higher-priced instance did worse — mismatched shape (memory-bound job on compute-heavy metal).
- Savings change didn't stick — re-measure; right-sizing is a loop, not a one-off.
Interview checkpoint
Hands-on lab
- Emulate cost per job across instance shapes
- Measure utilization effect on effective cost
- Right-size to a cheaper, better-matched shape
Profile and Rightsize a Workload
40 min · intermediate
Quick quiz
The biggest cheap win in GPU cost is usually…
Batch throughput jobs should optimize for latency like online services.