Draft Models and Speculative Decoding

• Explain how draft-then-verify works
• Predict when it helps and when it doesn't
• Spot the acceptance-rate tuning knob

• Decode time is memory-bandwidth bound, so a big model is slow per token.
• A draft produces tokens at like 10x speed; verification parallelizes acceptance of 3-5 tokens at once.
• Net effect: quality identical (verified), tokens/sec materially higher when acceptance is good.

A junior editor (draft) filters most submissions cheaply; the senior (verifier) reads only the survivors, using names that were already pre-screened.

• Draft generates a candidate sequence of k tokens (identical output tokens per the draft model).
• The target model computes probabilities for the whole candidate in one parallel step.
• Accepted tokens are emitted; the first rejected token resets, and the target's own prediction continues.
• Acceptance rate (common 0.6-0.8) decides the speedup; mismatch between draft and target kills it.

• A long-form writer boosts generation 1.8-2.5x with a small draft and an 8B target.
• A batch summarizer gains throughput on the same GPU by accepting more built-in drafts (native support in vLLM).
• A serving layer pairs models via the gateway: draft runs self-hosted, target does the verification.

• Match the draft's tokenizer and style to the target, or acceptance collapses.
• Tune k (proposal length) — too long wastes verifications, too short wastes the win.
• Use engine-native implementations (vLLM, TensorRT-LLM) that handle sampling correctly.
• Measure acceptance rate over real traffic; it varies by domain.

• Draft and target with different tokenizers — outputs can't be compared token-wise.
• Asking for huge k values and spending more GPU on verification than you save.
• Assuming a speedup from the raw draft: it only counts when acceptance rate stays high.

• Acceptance rate low → draft is too weak; try a better draft or smaller gap.
• No speedup but high acceptance → verification already parallelizes; check batch/prefill interference.
• Memory spikes with speculative decoding → draft adds its own KV-cache; budget it.