Autoregressive Generation
One token at a time, forever
Tokens are generated one at a time, each conditioned on everything before. This sequential dependency decides the shape of speed tricks (speculative decoding, KV cache), the ceiling on determinism, and the character of long generations.
What you will learn
- Explain why decoding is inherently sequential
- Trace where the KV cache helps
- Describe speculative decoding intuition
What is this?
Autoregressive generation predicts token t+1 from tokens 1..t, then feeds the result back in. Because each step depends on the last, decoding is sequential — you cannot parallelize what you haven't computed. This single constraint explains output latency, the KV cache, and the speed tricks like speculative decoding and draft models.
Why do you need it?
- Latency: a 500-token answer costs ~500 sequential forward-equivalents, not one.
- The KV cache makes repeated generation orders-of-magnitude cheaper per token than re-encoding the whole prompt each time.
- Sequentiality is why long outputs feel slow even on fast hardware.
- Draft-then-verify lets you check many candidates at once (parallel within the draft).
Mental model
Writing a novel in order
Autoregressive generation
You must write sentence 2 after sentence 1 exists, because it continues it. A fast typist doesn't get to draft all the sentences at once — but a good outline (a draft model) lets you validate whole paragraphs in parallel when they're plausible enough.
How it works
- Step 1: run the model on the prompt to get the first token.
- Cache is a key/value store for each attended token so attention doesn't recompute history.
- Step N: gather cached past + new token → predict token N+1 cheaply.
- Speculative decoding: a small model proposes K tokens; the big model validates all K in one parallel shot when they match.
- Output ends at an end-of-sequence token or the max-token cap.
Sequential decode with caching.
In production
- Every streaming answer is token-by-token — arrive early, refine as it goes.
- Long-document assistants lean on cached prompt KV to keep follow-ups fast.
- Latency budgets must assume tokens ≈ sequential calls unless you adopt draft models or caching tricks.
Best practices
- Stream answers so users see progress instead of one blocking blob.
- Reuse long static prompts via prompt/KV caching when the platform supports it.
- Match max_tokens to real needs — every cap is a real-time + cost decision.
Common mistakes
- Assuming 'generate 800 tokens' is one request-time unit.
- Ignoring that the first token is proportionally slow (prompt encoding).
- Turning off caching 'for safety' and paying for it in latency.
Common issues
First token latency feels hugeFull prompt must be encoded before the first decode
Cache the static prefix; reduce prompt size; move long context aside.
Steady streams stall periodicallyContext grows and attention cost rises per token
Trim/retrieve relevant context and summarize stale parts.
Debugging tips
- Slow first token but fast follow-ups — prompt-encoding bound; cache your static prefix.
- Slow every token — the model eval is the wall; consider smaller/draft paths.
- Timing spikes mid-stream — check for long tool arguments inlined into context.
Interview checkpoint
Quick quiz
Why is decoding sequential?
Speculative decoding improves quality of answers.