Inference Optimization
Intermediate 12 min read Inference Optimization

Quantization

Fewer bits per weight, same quality bar

Quantization stores weights in fewer bits (int8, int4) so the same GPU fits a bigger batch, a longer context or a larger model. It trades a little quality for speed and memory — always validated on your own evals, not assumed.

quantizationint8int4fp16AWQGPTQ
Prerequisites: GPU Inference Basics
Before we start

What you will learn

  • Relate weight bits to VRAM and speed
  • Choose a quantization scheme and level
  • Evaluate quality before and after
Introduction

What is this?

Quantization maps weight values into a lower-precision grid — fp16 (16-bit) to int8 or int4. That shrinks how much VRAM weights occupy and how much memory bandwidth each decode step consumes, which is exactly what makes inference slow. Quality drift is small for most tasks but must be checked.

Motivation

Why do you need it?

  • An 8B fp16 model is 16GB; int8 drops it to 8GB, int4 to ~4GB.
  • Faster decode: fewer bytes read per token means higher tokens/sec.
  • It can fit a bigger model or a longer context on an existing GPU.
Think about it

Mental model

Imagine

Quantization

Think of

A lossy photo

You keep the image recognizable at 10% of the file size; whether it's good enough depends on where the photo is used.

Mechanics

How it works

  • fp16 uses full precision; int8 rounds each weight to one of 256 steps around a learned scale.
  • int4 rounds to 16 steps; smarts (per-channel scales, group scaling) fight quality loss.
  • Quantization-aware training (QAT) nudges weights to quantize cleanly; post-training (PTQ) trades more quality for simplicity.
  • KV-cache can also be quantized (fp8) to extend the batch budget cheaply.

VRAM budget before and after.

fp16
int8
int4
Quality bar
Hands-on

Examples

Loading a quantized checkpoint

Loading a quantized checkpoint
# vLLM picks precision on weights
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq --dtype half \
  --max-model-len 8192
# or for the same card, more batch room:
vllm serve ... --quantization gptq_marlin \
  --max-num-seqs 256

The engine loads quantized weights at startup; the win shows up as larger batch or longer context per GPU.

Real world

In production

  • An 8B model at int8 serves a chat product on modest GPUs at the same speed as double cost.
  • A long-context RAG service uses int4 to fit 64K windows in VRAM.
  • A team validates int8 vs fp16 on 200 golden prompts before flipping.
Guidelines

Best practices

  • Run your eval set at fp16 and quantized levels; accept only deltas within tolerance.
  • Prefer engine-tuned schemes (AWQ/GPTQ via vLLM) over blanket int4 for quality.
  • Quantize the model, keep the tokenizer and decoding settings identical.
  • Document which checkpoints are quantized and how they were validated.
Watch out

Common mistakes

  • Flipping to int8 on a whim and blaming 'the model' when answers regress.
  • Quantizing weights but not checking effect on long-context behavior.
  • Believing int4 needs no evals because 'people say it's fine'.
When things break

Debugging tips

  • Math/logic regressions after quantizing → raise to int8 or QAT version.
  • Speed unchanged after quantizing → you're compute-bound (prefill) not memory-bound (decode).
  • Odd outputs only on long prompts → check KV-cache dtype / context-path artifacts.
Interviews

Interview checkpoint

Practice

Hands-on lab

VRAM and Precision Budgets
25 min · intermediate
In progress
Objectives
  • Compute weight footprint per precision
  • Add KV-cache and batch to the VRAM picture
  • Decide int8 vs int4 for a fixed GPU

VRAM and Precision Budgets

25 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The main speed win of quantization is what?

2

Quantization quality is identical across all models and tasks.

Go deeper

Official documentation

vLLM quantization
docs
Hugging Face quantization guide
docs