Transformers
Intermediate 14 min read Transformers

Attention Mechanism

Reading the whole room, selectively

Attention lets each token look at all other tokens and selectively pull information — no fixed window. It is the idea that makes LLMs work, and the reason transformers won.

attentionquerykeyvaluetransformersequence
Prerequisites: What a Neural Network Computes
Before we start

What you will learn

  • Describe queries, keys and values
  • Explain why attention replaces fixed windows
  • Trade attention's power against its cost
Introduction

What is this?

Attention computes, for each token, a weighted mix of every other token's value, where weights come from matching a query against each key. It lets the model 'pay attention to' the relevant context regardless of distance — the crucial upgrade over fixed-window RNNs. Tokens attend to the whole sequence; that selective mixing is how meaning flows in transformers.

Motivation

Why do you need it?

  • Any token can influence any other in one step — long-range dependencies are native.
  • Unlike recurrence, attention processes all positions in parallel, which is why GPUs love transformers.
  • The pattern is data-driven: the model learns WHAT to attend to for each task.
  • Nothing before it handled 'need to look far away' and 'still train fast' at once.
Think about it

Mental model

Imagine

A book club discussing a chapter

Think of

Attention

Every member (token) gets a say relevant to their interest. The query is your question, keys are each member's area of knowledge, values are what each member can contribute. You weigh contributions by how well the question matches their expertise.

Mechanics

How it works

  • Each token builds three vectors: query (Q), key (K), value (V), via learned projections.
  • Score = Q · K across all tokens; softmax normalizes the scores.
  • Output = weighted sum of all values using those weights.
  • High score = that token's value flows strongly into this token's representation.
  • Repeated over layers, attention composes deep-around references, not local ones.

One attention head on 3 tokens.

Tokens
Q/K/V
Scores
Weights
Output
Hands-on

Examples

Attention arithmetic in plain Python

Attention arithmetic in plain Python
import math
import numpy as np
 
tokens = np.array([[["it"], ["was"], ["it"]],
                   [[0.0], [0], [0]]])[0, :, 0]  # dummy tokens
x = np.array([[1.0, 0], [0.1, 0.9], [0.2, 0.8]])
# pretend projections = identity for this demo
q = x[0]; k = x; v = x
scores = np.array([np.dot(q, ki) for ki in k])
weights = np.exp(scores - max(scores)); weights /= weights.sum()
out = np.dot(weights, v)
print("weights:", np.round(weights, 2))
print("output :", np.round(out, 2))

The query 'it' settles an attention split — weights decide from which token the meaning arrives.

Real world

In production

  • LLMs rely on attention for summarization/translation/retrieval-heavy prompts.
  • Attention masks control causal vs bidirectional reading (see causal masking).
  • Attention's O(N^2) cost is THE scaling constraint in serving long contexts — KV caching exists to tame it.
Guidelines

Best practices

  • Remember attention cost grows with sequence length — budget context accordingly.
  • Rely on the framework's fused attention kernels; never reimplement manually.
  • Understand that attention tells you WHERE it looks, not WHY (interpretability caution).
Watch out

Common mistakes

  • Treating attention weights as a reliable explanation of model reasoning.
  • Queries without keys/values in the value mixing — attention without the V projection is just weighting.
  • Assuming all attention is contextual knowledge — much of it is positional/biweekly priors.
Root cause analysis

Common issues

Error
Bring-your-own attention is slow
Root cause

Naive O(N^2) implementation

Fix

Use fused/causal kernels or FlashAttention-style paths from the framework.

Error
Model ignores distant tokens entirely
Root cause

Attention learned a local bias or context too long

Fix

Evaluate on truncated context; consider windowed/relative attention.

When things break

Debugging tips

  • Tokens don't influence each other? Check causal mask interfering or attention head collapsed.
  • Long sequences slow — expect squared attention cost; switch to efficient variants if scaling.
  • An empty softmax (all extreme negatives) → logit scaling problem, usually numeric.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which vectors does a token emit?

2

Attention lets distant tokens directly influence each other.

Go deeper

Official documentation

Attention is All You Need
spec
The Illustrated Transformer
docs