Model Serving
Intermediate 12 min read Model Serving

Request Batching

The GPU's favorite way to multitask

Batching packs many prompts into one forward pass so a GPU amortizes memory movement across requests. Continuous batching joins requests mid-generation; contiguous batching blocks are the naive and slower baseline.

batchingcontinuous batchingthroughputGPU utilizationqueueing
Prerequisites: Providers vs Self-Hosting
Before we start

What you will learn

  • Explain why batch size drives throughput
  • Contrast static vs continuous batching
  • Tune batch size against latency
Introduction

What is this?

A GPU forward pass is far faster when many sequences run together, because the same weights are loaded once and reused across the batch. Batching groups concurrent requests so memory bandwidth is spent serving several requests at once instead of one at a time.

Motivation

Why do you need it?

  • Throughput (tokens/sec) scales near-linearly with batch size until memory runs out.
  • Latency goes up slightly; your batching policy is the throughput-vs-p99 trade.
  • Modern servers (vLLM) cursor-style batch at the token level — the single biggest serving win.
Think about it

Mental model

Imagine

Batching

Think of

Carrying groceries

One trip with many bags beats many trips with one bag; the limit is how much you can physically hold (memory).

Mechanics

How it works

  • Requests waiting in a queue are grouped and passed through the transformer once per batch step.
  • Static batching fills a fixed window: slow (long) requests block the batch to completion.
  • Continuous batching lets each sequence finish and leave as soon as it EOS-tags, freeing the slot.
  • max_num_seqs / batch window caps memory (KV-cache) per slot.

Continuous batching frees slots early.

Join
Mixed generation
Early exit
Refill
Hands-on

Examples

Throughput vs batch in one plot concept

Throughput vs batch in one plot concept
# pseudocode — the shape of the decision
for batch in scheduler.next_batches(max_tokens_or_quota):
    out = model(batch.prompts)          # one pass, many sequences
    for seq in batch:
        if seq.finished: batch.release(seq)
        else: batch.extend(seq.next_logits)

Continuous batching releases finished sequences mid-run, trading a little scheduler work for much higher GPU utilization.

Real world

In production

  • An assistant serves thousands of chats from one vLLM pod tuned with max_num_seqs 256.
  • An offline batch job packs 10k prompts at batch 64 to cut the bill by 80%.
  • A realtime path sets a small batch window to hold p99 under 1s.
Guidelines

Best practices

  • Raise batch size until p99 latency hits your budget; that's the throughput ceiling.
  • Use continuous batching servers; static batching wastes GPU on finished tails.
  • Queue with timeouts so batch wait is visible and bounded.
  • Leftpad inputs to identical lengths only if the server doesn't do it for you.
Watch out

Common mistakes

  • Treating 'batch 1' latency as the no-ops baseline — you're paying max per token.
  • Blowing the KV-cache scheduling memory and seeing OOM or thrash.
  • Measuring throughput without percentiles, hiding stragglers.
When things break

Debugging tips

  • Throughput flat as batch grows means you've hit memory (KV-cache), not compute.
  • p99 spike on one long prompt — check if continuous batching is actually enabled.
  • Queued-request timeouts pair with high batch: either is a tuning signal.
Interviews

Interview checkpoint

Practice

Hands-on lab

Simulate a Batch Scheduler
35 min · intermediate
In progress
Objectives
  • Model a token-level batch scheduler
  • Contrast static exit vs early exit
  • Compare finished-job counts and utilization

Simulate a Batch Scheduler

35 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What is continuous batching?

2

Larger batches always mean lower p99 latency.

Go deeper

Official documentation

vLLM continuous batching
docs
TensorRT-LLM batching
docs