Latency and Throughput Tuning
Finding the service's operating point on purpose
Tuning real-time serving means finding the latency/throughput operating point: measure the tail (p95/p99), then adjust batching, concurrency, model precision, and caching in deliberate experiments. Every change trades one axis for another — the discipline is measuring the trade.
What you will learn
- Measure and target tail latency, not just means
- Identify the levers: batching, concurrency, precision, cache
- Run tuning as controlled experiments with budgets
What is this?
Latency and throughput tuning adjusts serving parameters — batch windows, worker concurrency, model precision, caching — to hit a stated SLO: meet p95 for latency and maximize throughput per dollar. It's a series of measured experiments, not a one-time knob spin.
Why do you need it?
- Latency and throughput trade against each other: batching raises throughput but adds tail latency.
- Users feel the tail, procurement feels the cost, so both need deliberate targets.
- Precision and quantization cut latency materially when the accuracy hit is acceptable.
- A measured tuning record beats folklore ('GPUs are fast') for decisions.
Mental model
Tuning a race car for a specific track
Serving performance tuning
You don't 'make it fast' globally; you set gearing (batching), tire pressure (concurrency), brake bias (precision) for THIS track, then measure lap times (p95). A setup that wins the straights loses the corners.
How it works
- Establish a baseline: instrument p50/p95/p99 and throughput under a defined synthetic or replayed load.
- Identify the dominant cost: model forward, feature fetch, serialization, or queueing.
- Adjust one lever at a time — batching window, concurrency, dtype, cache TTL.
- Re-measure against the same load profile and the SLO.
- Record the change and result; keep the winning configuration as a config artifact.
The tuning loop.
In production
- A recommender gains 35% GPU utilization by raising the batch window from 4ms to 12ms while holding p95.
- Quantizing a vision model fp32→int8 cuts latency 2x with a 0.4% accuracy dip inside the SLO.
- A results cache absorbs repeated top-page requests, dropping server load dramatically.
Best practices
- Define the SLO pair (p95 and throughput target) before tuning.
- Measure with workload-realistic requests, not empty/synthetic-only.
- Change one lever at a time and record it like an experiment.
- Include an accuracy/cost budget when trading precision for speed.
Common mistakes
- Optimizing mean latency while the tail stays blown.
- Batching until p95 dies, then 'fixing' latency by more replicas at higher cost.
- Quantizing with no accuracy gate — speed at unknown quality cost.
- Tuning on load that isn't your production pattern.
Debugging tips
- p95 off target — draw the latency histogram; a bimodal tail usually flags cache hits vs cold joins, not the model.
- Throughput capped — check for a single shared lock or a serialize bottleneck in the handler.
- After quantization, odd errors — verify the runtime genuinely supports the dtype end-to-end.
Interview checkpoint
Hands-on lab
- Establish a latency baseline under simulated load
- Vary one lever and measure the p95 effect
- Choose a configuration inside the SLO
Run a Measured Tuning Experiment
40 min · advanced
Quick quiz
Tuning one lever at a time is important because…
Quantization is purely a speed lever with no correctness dimension.