Positional Encoding
Teaching order to a bag of words
Attention is position-agnostic by default — without position signals, 'the dog bit the man' equals 'the man bit the dog'. Positional encoding injects order, and its design constrains context length.
What you will learn
- Explain why position must be injected
- Contrast absolute vs relative encodings
- Connect encodings to context length limits
What is this?
Attention as defined is permutation-invariant: swap two tokens and the scores/computations don't see why order matters — but language does. Positional encoding adds a position-dependent signal, either added to embeddings (absolute, e.g. sinusoidal, learned) or injected into attention (relative, e.g. RoPE). Because positions must be representable, the encoding scheme ultimately bounds how far a model was trained to generalize.
Why do you need it?
- Syntax, negation and entailment are order-dependent; a model must know who did what to whom.
- Absolute encodings cap at max length; relative/rotary encodings extrapolate better.
- RoPE is the dominant modern choice, enabling the long-context extensions.
- Position in the KV cache is why efficient decode caches remain ordered.
Mental model
Numbering seats in a theater
Positional encoding
Row numbers (absolute) tell you where any seat IS, but for a group, relative spacing (who sits next to whom) captures interactions. Rotating coordinates is like assigning a continuous dial per seat so neighbors are close and far rows are far — preserving distance geometrically.
How it works
- Sinusoidal: fixed sine/cosine waves per dimension — relative offsets fall out naturally.
- Learned: an embedding table per position learned during training.
- Absolute: added to token embeddings before the first layer.
- RoPE: rotates Q/K by position angle so attention scores encode relative distance.
- Context length emerges from how far positions remain distinguishable.
When position is injected.
In production
- All decoder LMs ship with a positional scheme; you mostly inherit it via the checkpoint.
- Long-context tricks (context extension, position interpolation) modify the scheme, not the model.
- Choosing a checkpoint with RoPE helps future-proof very long prompts.
Best practices
- Stay within the checkpoint's trained position range — behavior degrades beyond it.
- Prefer RoPE-based checkpoints for long/dynamic input lengths.
- Treat 'context window' as 'trained positions', not a magic number.
Common mistakes
- Feeding 128k tokens to a model trained on 8k and expecting miracles.
- Assuming position doesn't matter for tool outputs/order in prompts.
- Writing own positional code without extrapolation testing.
Common issues
Quality nosedives past ~training lengthAbsolute position extrapolation breakdown
Use RoPE checkpoint, interpolate positions, or chunk the input.
Instructions at two positions conflictModel treats order as content contingent on position
Restructure the prompt to lead with high-priority items; keep strict order stable.
Debugging tips
- Model degrades suddenly past some length → you've left trained positions.
- Order-sensitive tasks failing (dates, steps) → suspect truncated/absent positional signal.
- Position counts feel wrong in later turns — the KV cache index vs mask row drift.
Interview checkpoint
Hands-on lab
- Implement the sinusoidal position formula
- Verify encodings are distinct per position
- Show how frequency varies across dimensions
Sinusoidal Positional Encodings
25 min · intermediate
Quick quiz
What problem does positional encoding solve?
A model trained on 8k positions works identically at 200k.