Transformers
Advanced 16 min read Transformers

Transformer Efficiency

Making quadratic cost affordable at scale

Quadratic attention, KV-cache growth and 4x FFNs make transformers expensive. Efficiency stacks at every level: fused attention kernels, KV caching, quantization, speculative decoding and memory planning.

KV cacheflash attentionquantizationspeculative decodingpagedAttentionthroughputlatency
Prerequisites: Decoder-Only (GPT) CNNs in Production
Before we start

What you will learn

  • Budget where transformer cost actually lives (attention + KV + FFN)
  • Use KV caching and fused attention to cut repeated work
  • Choose quantization and speculative decoding for generation
Introduction

What is this?

Transformer inference cost concentrates in three places: the O(T²) attention scores (mitigated by fused kernels), the KV cache that grows with context length (mitigated by paging/offload), and the 4x-wide FFN matmuls on every token. Efficiency work stacks optimizations across these: flash-attention avoids materializing T×T, KV caching makes per-token decode constant-time, int8/int4 quantization shrinks memory BW, and speculative decoding trades compute to skip serial decode steps.

Motivation

Why do you need it?

  • Cost grows slightly-quadratic in context and linearly in tokens — it dominates clinics of every LLM service.
  • Latency: each decode step is memory-bandwidth bound; quantizing weights 2x-4x shrinks it directly.
  • Throughput: batching + KV planning raises GPU utilization 2-10x.
  • Serving economics decide whether an LLM product exists at all — efficiency is a product decision.
Think about it

Mental model

Imagine

A concierge re-using one handwritten guest card instead of rewriting every time

Think of

KV caching

The concierge (model) once computes a guest's preferences (keys/values). Rather than recompute them for every followup request, they file and fetch the card — that's the KV cache, turning repeated context processing into a lookup.

Mechanics

How it works

  • KV cache: store past keys/values; each decode step computes only the NEW token's q and its attention against cached K/V.
  • FlashAttention: tiled, IO-aware attention — no big T×T matrix in HBM; ~2-4x speedups and linear-ish memory.
  • Quantization: int8/int4 weights (or KV) as lossy-but-cheap — less memory bandwidth per token.
  • Speculative decoding: a small draft model proposes k tokens; the big model verifies them in one parallel pass — net speedup ≈ draft acceptance.
  • Paged/continuous batching shares KV pages across requests; offloads to CPU/disk behind peaks.

Latency levers: KV cache, fusion, quantization, speculation.

Prefill
KV cache
FlashAttention
Quantized weights
Speculative decode
Real world

In production

  • Every serious LLM serving stack uses KV caching plus continuous batching.
  • vLLM/TGI serve with paged KV + int8/int4 quantized checkpoints as defaults.
  • On-device and cloud models alike merge speculative heads for 1.5-3x decode speedups.
Guidelines

Best practices

  • Profile: prefill (prompt compute) vs decode (memory-bound) budget separately.
  • Start with KV caches + a fused attention kernel; these are baseline, not optional.
  • Quantize with calibration and eval-validated quality; keep extended-precision embeddings.
  • Benchmark speculative draft acceptance — it's model-specific; expect 1.5-2x, measure it.
Watch out

Common mistakes

  • Recomputing the prompt's KV on every token (forgetting the cache exists).
  • Maximizing throughput while p95 latency explodes — define both.
  • Quantizing everything incl. embeddings/LayerNorms and losing quality needlessly.
  • Believing quadratic attention is 'handled' when your length is 4096+ and the kernel is naive.
Root cause analysis

Common issues

Error
Decode latency grows with conversation turns
Root cause

KV cache not retained across the session

Fix

Persist the cache for the conversation; reset only on context compact/truncate.

Error
Quantized model degrades serious accuracy
Root cause

Overshot int4 with no calibration or quantization-unaware sensitive layers

Fix

Calibrate on task data, keep first/last layers and embeddings in FP16, or use int8.

Error
AOT cache flushes under load
Root cause

KV paging policy evicts active sessions

Fix

Budget KV by session, use continuous batching, and prefetch/swap rather than drop.

When things break

Debugging tips

  • Flame-report prefill vs decode: >90% decode means memory-bound; >50% prefill means compute-bound — different fixes.
  • Record KV cache size in bytes as context grows — a linear curve reveals off-by-one in caching decisions.
  • After quantization, diff logits on a benchmark set vs FP16; large top-K flips signal a bad calibration.
  • A/B speculative decode on long documents — acceptance rate is the health metric.
Interviews

Interview checkpoint

Practice

Hands-on lab

Budget KV Cache Memory and Quantization
30 min · advanced
In progress
Objectives
  • Compute KV cache bytes as a function of context length
  • Measure quantization's effect on weight bytes and a proxy quality signal
  • Derive the served-tokens-per-site economics equation

Budget KV Cache Memory and Quantization

30 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What is the KV cache?

2

Flash attention avoids storing the full T×T attention matrix in main memory.

Go deeper

Official documentation

FlashAttention paper
docs
vLLM docs
docs