Transformers
Intermediate 15 min read Transformers

Attention

Soft alignment: where to look when producing each token

Attention lets a model weight how much each input token matters when producing each output token. It replaced fixed-size context vectors and fixed recurrence, and it is the core of transformers.

attentioncontextalignmentweighted averageencoder-decodermechanism
Prerequisites: NLP LSTM
Before we start

What you will learn

  • Explain attention as a weighted average over inputs
  • Compute attention weights from a query and keys
  • Explain why attention beats recurrent context for long sequences
Introduction

What is this?

Attention computes, for each output position, a weighted sum over all input positions — the weights come from a query interacting with each input's key. Instead of squeezing a whole sentence into one vector (or forcing order through a recurrence), the model can 'look at' whichever inputs matter per step. This decouples context size from any fixed bottleneck and is parallelizable, which is why attention became the backbone of modern NLP.

Motivation

Why do you need it?

  • A single context vector loses detail on long inputs; attention reads the whole sequence directly.
  • Alignment for translation: 'dog' steps should watch the French word for 'dog', not just the summary.
  • No recurrence → all positions processed in parallel, enabling scale to hundreds of billions of tokens.
  • It's differentiable, so alignment is learned end-to-end with gradients, not engineered.
Think about it

Mental model

Imagine

A student reading exam notes per question

Think of

Attention at decoding time

For the 'What caused the crash?' question the student (query) scans the notes (keys) and reads hardest the pages on brakes and ice (values); for another question they re-read the schedule section. Different queries, different weighting — the same notes, no rewriting.

Mechanics

How it works

  • Each input token has a key k and a value v; the output position has a query q.
  • Score = q·k for each input — a similarity dot product.
  • Weights = softmax(scores): positive, sum to 1, concentrated on relevant inputs.
  • Output = Σ weights_i · v_i — a weighted average of the values.
  • Layer output: a context-dependent reading of the sequence at this position.

Query attends over keys to weight values.

Queries
Keys
Scores
Softmax
Weighted values
Real world

In production

  • Encoder-decoder translation: decoder tokens attend over encoder states — the original attention application.
  • Cross-attention in multimodal models lets image patches be 'read' when producing text.
  • Retrieval-augmented generation weights over retrieved chunks at generation time.
Guidelines

Best practices

  • Scale dot products by 1/√d_k to keep softmax temperatures stable at large dimensions.
  • Use multiple heads (multi-head attention) so different subspaces can attend differently.
  • Mask future positions (causal) in decoders so generation never peeks.
  • Attend over keys/values only; keep scores normalized for interpretable heatmaps.
Watch out

Common mistakes

  • Forgetting the scale factor (1/√d) — attention with huge dims saturates into one-hot.
  • Applying attention without masking in autoregressive decoding — trains that peek.
  • Reading attention as semantics — it's a learned weight, not an explanation.
  • Using attention on non-sequential bag-of-features where position matters but no encodings exist.
Root cause analysis

Common issues

Error
Attention weights collapse to near uniform
Root cause

Scale factor omitted or model dimension too large without 1/√d

Fix

Divide scores by √d_k; warm up LR so softmax doesn't start saturated.

Error
Model attends off-diagonal (wrong tokens) consistently
Root cause

Mis-aligned key/value projection or positional encoding stripped

Fix

Verify attention matrix orientation (q attends to k) and keep positional info.

Error
Attention matrix is all on [PAD]
Root cause

Padding mask missing or applied to wrong tokens

Fix

Mask pad positions additively (−∞) before softmax and add to the loss mask too.

When things break

Debugging tips

  • Print attention weights as a heatmap: near-one-hot means scaling/saturation; near-uniform means dense collapse.
  • Compare a 100-token vs 1,000-token input — attention should stay O(T) script-only, not degrade.
  • Overfit a toy 'position' task (attend to the 3rd token) to validate the machinery.
  • Find straight-line attention (all weight on one token) — often a scaling or head-collapse bug.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Attention weights over inputs are computed by:

2

A decoder's attention must be masked so it can't peek at future tokens.

Go deeper

Official documentation

Attention Is All You Need
docs