Scaling Serving
Right-sizing GPUs to token demand
Serving scales on tokens and concurrency, not requests: your GPU count follows tokens/sec demand times the batch-memory envelope. Autoscaling, queue depth and replica warm-up decide whether scaling feels instant or sluggish.
What you will learn
- Model capacity as tokens/sec × max concurrency
- Design autoscaling that doesn't chase every spike
- Budget cold-start into scaling policy
What is this?
Serving capacity is throughput, not request count: requests with long outputs eat far more KV-cache and decode time than short ones. Scaling means matching replica count to sustained token demand while keeping cold-start (model load) inside your latency budget.
Why do you need it?
- One GPU with 8K contexts and 8 max sequences handles far fewer chat turns than 256 max_seq prompting 10-turn sessions.
- Autoscaling on requests misjudges cost: a burst of 100 long generations needs 10x the GPUs of 100 short ones.
- Model load takes seconds-to-minutes; a cold replica can't serve its first burst.
Mental model
Scaling
Staffing a restaurant
You staff to covers-per-hour and kitchen throughput, not to the number of customers in the room at once — some orders are 12 courses, some are a coffee.
How it works
- Each replica has a max concurrent sequences and a token ceiling via its KV-cache budget.
- A queue holds overflow; too-deep queues push latency past budget.
- Autoscaler reacts to queue depth and utilization, adding replicas ahead of RAM saturation.
- Warm pools or preloaded models keep cold-start off the critical path.
Queue → replicas loop.
In production
- A chat service keeps 2 warm replicas and scales to 8 on queue-depth during office hours.
- A batch job bursts 50 replicas overnight, down to 1 at dawn.
- A platform preloads two model flavors per node so flavor swaps don't cold-start.
Best practices
- Autoscale on queue depth and tokens/sec, not request count.
- Keep a warm spare always loaded in critical pools.
- Bound queue depth with a timeout so saturation surfaces as errors, not silent p99.
- Pin replicas to one model to avoid heterogeneous cold starts.
Common mistakes
- Autoscaling to instant spikes: cold load lags, so you scale down before the new pods are warm.
- Oversizing for the tail and paying idle GPU.
- No max on replicas: an attack or runaway job multiplies your bill.
Debugging tips
- Sustained p99 but healthy p50 = a straggler row or a skewed batch, not capacity.
- Queue grows but replicas idle = scheduler/autoscaler config, not GPU.
- Cold-start in the metric timeline confirms the autoscaler lagged demand.
Interview checkpoint
Quick quiz
Best autoscaling signal for LLM serving?
A cold replica can immediately serve a traffic spike.