Model Serving
Intermediate 12 min read Model Serving

GPU Inference Basics

Why VRAM and bandwidth rule the cost

Inference on GPUs is memory-bandwidth bound: every generated token needs the full weight set read from VRAM once. That fact explains why model size, KV-cache and quantization dominate your serving math.

GPUVRAMKV cachememory bandwidthinference
Prerequisites: Request Batching
Before we start

What you will learn

  • Estimate VRAM for a model and its KV-cache
  • Explain why inference is memory-bound
  • Read a GPU profiler for serving
Introduction

What is this?

Transformer inference repeats one pattern for every token: read all weights from high-bandwidth memory and do a small amount of compute per read. At low batch sizes the GPU is starved waiting on memory, which is why VRAM size and bandwidth, not FLOPs, decide how a serving rig behaves.

Motivation

Why do you need it?

  • A 7B model at fp16 is 14GB of weights; your whole batch's KV-cache sits on top.
  • Smaller memory footprint (quantization) means bigger batches or longer contexts on the same GPU.
  • It explains why 2x GPU VRAM changes your cost more than 2x clock speed.
Think about it

Mental model

Imagine

GPU inference

Think of

Reading a book aloud

Each token is a page-read of the whole book (weights) from a shelf (VRAM); the bottleneck is walking to the shelf, not the reading speed.

Mechanics

How it works

  • Weights live in VRAM; each forward pass streams them through compute units once per token.
  • KV-cache stores the running attention state, growing with batch size × context length.
  • Quantization reduces weight bits (fp16→int8→int4), shrinking reads at some quality cost.
  • Prefill (prompt) is compute-heavy; decode (token-by-token) is memory-heavy.

Where VRAM goes.

Weights
KV-cache
Activations
Fragmentation
Real world

In production

  • An 8B model at int8 packs into one 24GB GPU with room for 8K contexts.
  • A long-context app reserves KV-cache headroom, capping batch to fit 32K windows.
  • A platform shards a 70B across 2 GPUs using tensor parallelism for larger single requests.
Guidelines

Best practices

  • Know the formula: VRAM ≈ weights + batch×len×layers×2 floats for KV-cache.
  • Choose precision deliberately: int8 for the default, int4 only after evals.
  • Cap max context before serving; the KV-cache grows per prompt+output tokens.
  • Enable paged/greedy schedulers (vLLM, TGI) to cut fragmentation.
Watch out

Common mistakes

  • Buying a GPU on flops and maxing out on VRAM — the actual bottleneck.
  • Enabling long contexts nobody uses, raising KV-cache and tanking batch.
  • Quantizing without running quality evals, then blaming 'the model' for regressions.
When things break

Debugging tips

  • nvtop shows 100% memory but low compute = memory-bound decode; add batch, not flops.
  • OOM at high concurrency = KV-cache budget, not model weights. Check max_num_seqs.
  • Prefill slow but decode fine = compute-bound prefill — consider splitting or smaller model.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

At decode time, the GPU is usually bound by what?

2

Quantization mostly reduces VRAM and makes batching easier.

Go deeper

Official documentation

vLLM KV cache engine
docs
NVIDIA GPU memory guide
docs