Scaling and Autoscaling
Capacity that reacts to demand, not guesswork
Serving scaling is capacity planning metric-driven: the binding resource (CPU, memory, GPU) sets the ceiling, and autoscaling decisions react to utilization, queue depth and traffic. Autoscale isn't automatic-to-good — it's configured by deliberate signals.
What you will learn
- Choose the metric an autoscaler should follow
- Weigh horizontal vs vertical scaling per workload
- Set bounds so autoscaling can't bankrupt or under-serve
What is this?
Scaling adjusts serving capacity to demand. Horizontal scaling adds/removes replicas (whole servers); vertical scaling resizes the instance. Autoscaling automates those decisions on metrics — utilization, queue depth, request rate — within configured floors and ceilings so a burst grows capacity and a lull shrinks cost.
Why do you need it?
- Static capacity either over-provisions (paying for idle) or under-provisions (latency spikes).
- GPU workloads have a hard utilization ceiling that replicas alone can't beat.
- Autoscaling on the right metric turns demand into capacity with lag, not panic.
- Bounds keep autoscaling honest: a runaway can't empty the budget.
Mental model
A call center adding agents at peak and reassigning at lull
Autoscaling replicas
The supervisor (autoscaler) watches hold queue (queue depth) and per-agent busy-ness (utilization), hires (scales up) with a lag while holding queued calls, and re-assigns agents (scales down) — but never below the skeleton crew (min replicas).
How it works
- You define metric(s): CPU, memory, GPU utilization, queue depth, rps.
- The autoscaler samples and evaluates against a target (e.g. 70% CPU).
- If needed, it changes replica count — bounded by min/max.
- New replicas take time to be ready (image pull, model load), so scale-up lag matters.
- Scale-down is rate-limited to avoid thrash under oscillation.
The autoscaling feedback loop.
In production
- A recommender horizontals on request queue depth, adding replicas before latency dies.
- A GPU model scales within 1-3 replicas because a GPU is a hard capacity unit.
- A quiet batch service scales to zero at night via a cron-timed minimum.
Best practices
- Autoscale on the metric that's actually binding (GPU util for GPU models, queue for IO).
- Set meaningful min/max; the floor protects latency, the ceiling protects budget.
- Account for model-load time — warm new replicas before your scale-up bottleneck arrives.
- Cooldown and stabilization windows prevent oscillation.
Common mistakes
- Autoscaling on CPU when the model is GPU-bound — the signal ignores the real bottleneck.
- No max replicas; a spike bills you a fleet.
- Scale-to-zero on a service whose first-call latency matters.
- Replicas that all pull the huge model at once, causing a thundering herd on cold start.
Debugging tips
- Scaling uselessly — check the metric used; a wrong metric has no relationship to demand.
- Scale lurching (up/down) — widen stabilization windows; identify metric oscillation.
- New replicas useless instantly — model-load dominates readiness; look at readiness probe gates.
Interview checkpoint
Hands-on lab
- Simulate demand against a capacity model
- Choose and tune autoscaling signals
- Assess scale-lag and inflection points
Autoscale on the Right Signal
40 min · advanced
Quick quiz
Which signal should a GPU model service autoscale on?
Min/max replica bounds exist to protect both latency and budget.