ML Workload Profiling

• Instrument training and serving workloads for profiling
• Identify utilization, memory, and I/O bottlenecks
• Turn profile findings into cost decisions

• You can't right-size a workload you've never measured.
• Most 'GPU problems' are data or scheduling problems wearing GPU glasses.
• Profiling exposes idle money: low utilization, oversized reservation, wasted checkpoint churn.
• Profiles turn cost meetings from opinion into evidence.

You could fit the stadium by feel, or you can time every heat, watch pulse (utilization), and count the refresh stops (stalls). The evidence says which runner (workload) to train harder and which lane (shape) they actually need.

• Pick the profile: training run, or serve-time latency under load.
• Instrument: GPU% and memory over time, load stalls, I/O and checkpoint duration, latency p50/p99.
• Correlate: what was happening (data load? sync? queue) at each low-utilization interval.
• Model it: replace the 'big number' with a per-phase decimal balance — train, eval, checkpoint.
• Act: right-size shape, tune batch, move I/O off critical path, then re-profile.

• A training team's profiler finds 55% of time is eval+checkpoint, not training — they shrink eval frequency and trim checkpoints.
• A serving team's latency profile under load reveals a GPU stall per request — data serialization, fixed, not instance size.
• A profiling run shows memory is 20% of the box — the team downsizes 3 instances and keeps SLO.

• Profile the real workload under representative load — not a dimmed toy.
• Automate a lightweight profile into each release pipeline.
• Report profiles in cost reviews; 'here's the measurement' beats 'I feel like'.
• Re-profile after every tuning change; profiles decay as data and traffic shift.

• Profiling during lunch-hour logs and calling it 'the workload'.
• Ignoring memory, staring only at GPU clocks.
• Treating profile and cost as separate silos (both just hang on the same numbers).
• Optimizing a phase that's 3% of the runtime (spending hours to shave nothing).

• GPU% low while waiting on logs — I/O is the bottleneck; prefetch/augment off the critical path.
• Memory pinned at 99% — either correctly dense or shape-mismatched; check the memory/compute weighting.
• Latency p99 swings while p50 is fine — one noisy phase (serialization, cold start) dominates; isolate it.