Autoscaling ML Workloads
Replicas that follow demand and queue depth
Autoscaling on K8s spans serving (HPA/KEDA scales replicas on traffic and queue metrics) and batch/training (scale workers and bins). The tricky part is GPU-quantized capacity, model-load warm-up, and scale-to-zero latency tradeoffs.
What you will learn
- Configure HPA on the right metric for ML services
- Scale on queue/event metrics with KEDA
- Handle GPU-quantized scaling and warm-up lag
What is this?
Autoscaling for ML workloads scales serving replicas and batch parallelism from real signals — CPU/GPU utilization, rps, and queue/event depth via KEDA or HPA custom metrics — within configured floors and ceilings that respect GPU quantization and model-load warm-up.
Why do you need it?
- Serving demand is spiky; static fleets both overpay and under-serve.
- Metric choice decides signal quality — queue depth predicts pain better than CPU%.
- Scale-to-zero saves real money for batch and quiet services.
- K8s autoscaling is native; ML's job is choosing signals and bounds well.
Mental model
Ride-share dispatching: idle cars scale out at rush, in at lull
HPA/KEDA autoscaling
The dispatcher watches rider demand (queue) and adds cars — but only up to the fleet cap (max) and never below the airport standby floor (min). A car joining the zone is a capacity pod whose warm-up is the drive time to the pickup.
How it works
- HPA reads metrics (CPU, custom/Prometheus) and reconciles replica count toward the target.
- KEDA extends to queue/event sources (Kafka, Rabbit) scaling on backlog depth.
- Scale-down is stabilized/cooldowned so oscillation doesn't thrash replicas.
- GPU-based services scale in GPU-quantized steps and account for model-load time on new pods.
- Scale-to-zero parks capacity when idle and pays a cold-start on wake.
Autoscaling signals and action.
In production
- A KEDA-triggered consumer scales out as a Kafka topic backlog climbs.
- An HPA scales a recommender on rps while GPU-bound pods scale in whole-GPU steps.
- Scale-to-zero turns a nightly scoring service into near-zero off-peak spend.
Best practices
- Autoscale on the metric that predicts demand — queue depth over CPU for many workloads.
- Set min replicas for latency-critical service; scale-to-zero elsewhere deliberately.
- Add stabilization/cooldown to kill oscillator thrash.
- Account for model-load time in readiness and min replicas; warm new capacity.
Common mistakes
- Autoscaling a GPU service on CPU% — signal unrelated to the bottleneck.
- Scale-to-zero for interactive latency, eating a cold start per user.
- No stabilization — replica flapping between 2 and 20 every few minutes.
- New replicas that sniff readiness for 40s+ of model load, so scale-up is always too late.
Debugging tips
- Autoscaler ignores traffic — wrong metric selection feeding inactive targets.
- Scale churns — inspect the metric time-series and widen stabilization.
- Scale-to-zero wakeups are slow — pre-warm or raise min replicas; measure cold path latency.
Interview checkpoint
Hands-on lab
- Model a metric-to-replica controller
- Show queue-depth driven scaling beats CPU-only
- Apply stabilization to kill oscillation
Autoscale on a Queue Signal
35 min · advanced
Quick quiz
Scale-to-zero is appropriate when…
Stabilization windows prevent autoscaling oscillation.