Transformers
Intermediate 16 min read Transformers

Decoder-Only Transformers (GPT)

Predict the next token, forever

GPT stacks causal decoder blocks and pretrains by predicting the next token. Same architecture, no encoder — and with scale it became the base of modern LLMs and agents.

GPTdecoder-onlyautoregressivenext-token predictioncausal maskLLM
Prerequisites: Transformer Architecture
Before we start

What you will learn

  • Explain causal masking and next-token pretraining
  • Describe inference as repeated token-by-token sampling with KV cache
  • Outline why scale turned next-token prediction into a general engine
Introduction

What is this?

A decoder-only transformer is the same block stack with strictly causal self-attention: token t may only attend to tokens ≤ t. It is pretrained by predicting each next token given the prefix (max-likelihood over the corpus). At inference you loop: pass the (cached) sequence, read the final token's logits, sample a token, append it. Scale — lots of parameters, data and compute — turned this simple objective into the modern LLM and agent engine.

Motivation

Why do you need it?

  • No encoder → simple, uniform, easily scaled training pipeline.
  • Causality makes it a proper probability model over token sequences — native for generation.
  • The same model handles classification and chat when prompted: tasks are just sequences.
  • Scale plus the mask is all 'emergent' ability really is — an extremely large next-token model.
Think about it

Mental model

Imagine

An autocomplete engine with a photographic memory

Think of

A decoder-only LLM

Every keypress suggests the most likely continuation. Train on nearly all of humanity's written text and the suggestions become paragraphs, essays, plans. It never 'plans a task'; it just always knows what text plausibly comes next — which for many tasks is the answer.

Mechanics

How it works

  • Tokenize input, add (learned/Rotary) position, stack causal blocks.
  • Pretrain: cross-entropy between predicted next-token distributions and the actual next token on trillions of tokens.
  • Inference: keep the prefix's KV cache; per new token only compute the last position's Q/K/V path.
  • Decode: logits → softmax → sample (temperature, top-p or greedy) → append → loop.
  • Prompting is everything: tasks map onto the sequence — translate, paraphrase, answer — so no architecture change is needed.

Causal decode loop with KV cache.

Prefix tokens
Causal blocks
Sample
Append token
Next step
Real world

In production

  • Chat assistants, code completion, summarization — all decoder-only next-token models.
  • Agents: plan → act → observe encoded as a prompt loop.
  • Embeddings/evaluators: pool the last layer for classification heads on top.
Guidelines

Best practices

  • Sample with tuned temperature/top-p rather than always argmax (repetition vs diversity).
  • Cache K/V aggressively for long generations — it's the difference between O(T²) and O(T).
  • Match trained tokenization exactly; token counting drives context budgeting.
  • Set stop sequences explicitly (EOS plus custom stops) to avoid runaway generation.
Watch out

Common mistakes

  • Using a decoder-only model where bidirectional context would help (document classification benefits from BERT-style).
  • Forgetting the causal mask and letting training peek — silently learned impossible pattern.
  • Repetition loops from greedy decoding on low-entropy heads.
  • Assuming 'LLM = reasoning' — it predicts text; the reasoning you want lives in prompts and scaffolding.
Root cause analysis

Common issues

Error
Model repeats the same phrase forever
Root cause

Greedy sampling on a low-diversity head (repetition bias) or poorly-tuned stop sequence

Fix

Sample with temperature 0.7-1.0 + top-p, add repetition penalty, and enforce stop tokens.

Error
Generations ignore the prompt after a few tokens
Root cause

Long-prompt truncation or KV-cache mismatch after eager recomputation

Fix

Verify the prompt is fully within the context window and the cache builds incrementally.

Error
Training diverges fast (decoder-only)
Root cause

Missing warm-up/scale or pre-norm not applied

Fix

Pre-norm architecture, LR warm-up, gradient clipping, and scaled init (GPT-2 style).

When things break

Debugging tips

  • Check attention mask is strictly causal (assert triu is -inf).
  • Perplexity on a held-out text set as the pretraining health signal.
  • Sample with fixed seed for reproducible troubleshooting runs.
  • If output stalls/repeats: inspect logit entropy at the stuck position and stop-token behavior.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The causal mask in a decoder-only model:

2

During inference, recomputing the full prefix per token is efficient.

Go deeper

Official documentation

GPT-3 paper
docs