Capacity Planning for Inference

• Estimate serving throughput from concurrency and batch
• Size memory and GPU count for a workload
• Design overflow, throttle and fallback for spikes

• Under-provisioning = queueing or 503s at peak; over-provisioning = idle GPUs billed all month.
• KV cache, not just model weights, decides how many concurrent users a GPU handles.
• Spike response is a design decision now: throttle politely vs fail loudly.

You size it for worst realistic weekday traffic, not the Monday-morning record — and you build the ramp restrictions (throttling) so an occasional overload bends the bridge instead of breaking it.

• Find max concurrent sessions per GPU: KV-cache memory + batch slices per model.
• Measure batch-fill: bigger batches raise tokens/sec but add TTFT jitter.
• Project from usage: peak tokens/sec = sessions x tokens per request x burst factor.
• Add overflow: queue + provider fallback or graceful degradation (cached answers, smaller model).

• A serving team predicts 3 requests/sec x 2K tokens → sizes N GPUs at 60% fill before a launch.
• A chat provider rate-limits per tenant and spills overflow to a batch API at 5x traffic.
• An autoscaler scales on active sessions and p99 TTFT, not CPU.

• Measure batch-fill vs TTFT trade for your model, don't guess.
• Plan overflow behavior explicitly: queue depth caps, throttling responses, cached fallbacks.
• Track tokens/sec per GPU and per provider as the real capacity metric.
• Stress-test the overflow path before the launch spike, every quarter.

• Sizing GPUs by raw model weight only — KV cache is the real constraint.
• Autoscaling on request count while batch fills stay flat (mismatch with tokens).
• No degradation plan: the spike either times out silently or 500s.

• TTFT rockets at peak → batch too large at high fill; cap batch or split across shards.
• GPU not at memory limit but slow → bandwidth/KV pressure; check tokens/sec per unit.
• Spike 503s → overflow path misconfigured; verify queue and provider failover actually engage.