Inference Optimization
Intermediate 12 min read Inference Optimization

Throughput Tuning

Packing more tokens through the GPU

Throughput is tokens per second across the fleet, tuned by batch size, KV-cache budget, request concurrency and model precision. High throughput usually costs p99 latency, so you tune to a point, not to maximum.

throughputtokens per secondconcurrencybatchsaturation
Prerequisites: Request Batching
Before we start

What you will learn

  • Compute tokens/sec throughput correctly
  • Identify the binding constraint (memory, compute, queue)
  • Trade throughput against latency deliberately
Introduction

What is this?

Throughput tuning maximizes tokens generated per second per dollar. The levers, in order of impact: fill the batch (decode is memory-bound), enlarge max-concurrent-sequences up to the KV-cache ceiling, choose precision (int8/int4 shrink reads), and only then buy more GPUs.

Motivation

Why do you need it?

  • Under-batched GPUs are under-utilized; you paid for memory bandwidth you aren't using.
  • Throughput and p99 latency pull opposite directions — the right target is cost per token at an SLO.
  • Offline (batch) workloads can chase pure throughput; interactive ones cannot.
Think about it

Mental model

Imagine

Throughput tuning

Think of

Highway lanes

More lanes (concurrency) move more cars until the toll plaza (KV-cache) is the choke; adding lanes past that point only adds toll delay.

Mechanics

How it works

  • Saturate the card: raise max_num_seqs until memory (KV-cache) or latency caps out.
  • Measure tokens/sec with a fixed prompt+output mix, at the concurrency you'll actually serve.
  • Each further squeeze is measured, not assumed: quantization, fewer layers (smaller model), prefix caching.
  • Plot throughput vs p99; pick the knee that meets the SLO at least cost.

Throughput knee curve.

Idle
Scaling
Knee
Overshoot
Hands-on

Examples

Measure the knee

Measure the knee
# simulated: tokens/sec vs concurrent requests
for conc in [1, 4, 16, 64, 128]:
    tp = 4000 * conc / max(conc, 8)   # saturating shape
    p99 = 500 + 40 * conc
    print(conc, 'tps', round(tp), 'p99ms', p99)

Throughput flattens while p99 climbs — the knee is where you stop adding concurrency.

Real world

In production

  • An offline summarizer runs at concurrency 64 and trims cost per page.
  • An interactive assistant stays at concurrency 8 to hold p99 TTFT 900ms.
  • A batch pipeline routes to a dedicated (higher-concurrency) pool vs the realtime pool.
Guidelines

Best practices

  • Baseline tokens/sec at a fixed workload before any change.
  • Split realtime (low batch) and batch (high batch) pools.
  • Watch GPU memory vs utilization: 100% mem + low util = memory-bound decode, raise batch.
  • Charge users by tokens, and tune where the most tokens flow.
Watch out

Common mistakes

  • Chasing peak throughput on a realtime path and blowing TTFT.
  • Measuring with toy prompts that don't reflect your average output length.
  • Ignoring the queue: throughput at the model isn't throughput an app sees.
When things break

Debugging tips

  • GPU util low but memory high: increase batch/concurrency.
  • GPU util high but tps flat: compute-bound — smaller model or lower precision.
  • p99 climbs while tps flat: you're queueing, not serving — scale out.
Interviews

Interview checkpoint

Practice

Hands-on lab

Find the Throughput Knee
30 min · intermediate
In progress
Objectives
  • Sweep concurrency and observe throughput
  • Plot the memory/latency knee
  • Pick an operating point inside the SLO

Find the Throughput Knee

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Throughput is best measured in what unit?

2

Higher concurrency always raises throughput.

Go deeper

Official documentation

vLLM performance guide
docs
Measuring LLM throughput
docs