Transformers
Intermediate 14 min read Transformers

Positional Encoding

Teaching order to a permutation-invariant mechanism

Self-attention is order-agnostic — shuffle the input and scores are identical. Positional encodings inject order as (learned or sinusoidal) vectors added to token embeddings so the model can know position.

positional encodingposition embeddingsinusoidallearned PEropeattention order
Prerequisites: Self-Attention & Multi-Head
Before we start

What you will learn

  • Explain why self-attention needs position information
  • Compare sinusoidal vs learned positional encodings
  • Know when RoPE-style relative encodings win
Introduction

What is this?

Self-attention's QKᵀ similarity is invariant to shuffling token positions — 'the cat sat' and 'sat the cat' give the same attention pattern. Positional encodings break this by adding position vectors to token embeddings before attention (absolute) or by biasing scores by relative distance (relative, e.g. RoPE). Sine waves work out-of-the-box for unseen lengths; learned embeddings adapt to the corpus; RoPE encodes relative offsets directly into rotations.

Motivation

Why do you need it?

  • Without position, the model can't know word order — it defaults to a word bag.
  • Position is tightly linked to syntax and meaning ('2 days ago' vs 'two day') — the model must see it.
  • Length generalization differs by encoding type — finite-range learned PEs fail on longer contexts.
  • Relative encodings generalize length better and are now standard in modern LLMs.
Think about it

Mental model

Imagine

Seat numbers on a bus with assigned rows

Think of

Positional encodings

All passengers have identical boarding passes (identical embeddings); the seat numbers (position vectors) wired into their passes are what let the conductor know arrival order and who sits where. Sinusoidal numbers work for any bus length; learned numbers only for seen seat counts.

Mechanics

How it works

  • Sinusoidal: PE(t, 2i) = sin(t / 10000^(2i/d)), PE(t, 2i+1) = cos(...) — a dense fingerprint of position t.
  • Learned: a lookup table of T rows × d columns learned with the model; simple, corpus-tuned.
  • Add (not concat): token embedding + position vector, usually before the first block.
  • Relative (RoPE): apply a rotation that depends on the distance to each key — score naturally decays/repeats by relative position.
  • Choice impacts extrapolation: sinusoidal/some relative encodings generalize beyond training length; learned absolute typically don't.

Token embedding plus position vector into attention.

Tokens
Position
Add
Q/K/V project
Attention
Real world

In production

  • BERT/GPT use learned absolute position embeddings (fixed max length).
  • Modern LLMs (Llama, Mistral) use RoPE — rotation-based relative position.
  • T5/ALiBi use attention-bias style position for length extrapolation.
Guidelines

Best practices

  • Learn position as embeddings for a fixed max-length task (simplest, strong).
  • Prefer relative/rope when you need length generalization beyond training.
  • Keep position embeddings within the same scale as token embeddings (add, don't dominate).
  • Test extrapolation explicitly: eval on a longer sequence than any training sample.
Watch out

Common mistakes

  • Adding position AFTER the first block — it must ride into attention.
  • Using an un-learned/random sine generator on a fixed-length corpus when learned PE would win.
  • Softmax-then-add tricks that break the additive identity of position.
  • Ignoring ALiBi/RoPE when serving variable-length streaming decode.
Root cause analysis

Common issues

Error
Model treats the first and last token the same regardless of order
Root cause

Positional injection missing or inadvertently dropped

Fix

Add position embeddings before Q/K/V projection and verify through a controlled order test.

Error
OOD length evaluation collapses
Root cause

Learned absolute positions with no slot for longer inputs

Fix

Switch to RoPE or ALiBi, or pad-train with position shifting.

Error
Position dominates semantics
Root cause

PE scale much larger than token embeddings

Fix

Scale position vectors to a fraction of the embedding norm and re-validate.

When things break

Debugging tips

  • Compute attention between 'token A at pos 0' and itself copied at pos 5 — no position means identical scores.
  • Print PE matrix: sinusoids of different frequencies should differ visibly by row.
  • Level test: eval a trained model at 2x its max training position — learnable PE predicts its last row repeatedly.
  • If fine-tuning on longer text degrades, your absolute PE ran out of rows.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build and Inspect Positional Encodings
25 min · intermediate
In progress
Objectives
  • Generate sinusoidal positional encodings in numpy
  • Verify PE differentiates positions and scales sensibly
  • Measure a learned-PE failure on out-of-range positions

Build and Inspect Positional Encodings

25 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Positional information is added to the model's input as:

2

Learned absolute positional embeddings generalize well to lengths unseen at training.

Go deeper

Official documentation

RoFormer: rotary position embeddings
docs