Self-Attention
Tokens meeting internally
Self-attention is attention where every position IS both attendee and attended — tokens find each other within the same sequence. It is the workhorse layer inside every transformer block.
What you will learn
- Explain self-attention vs cross-attention
- Track how context shapes a token's representation
- Identify what a head might be 'doing'
What is this?
Self-attention runs attention inside a single sequence: each token's query attends to the keys/values of all other tokens (including itself). The result is a context-aware rewrite of every token. In LLM blocks, self-attention runs on the prompt tokens — this is precisely how the model 'reads' the whole prompt while writing each continuation.
Why do you need it?
- Copies resolution ('the llama ate its lunch') happens by one token pulling meaning from another.
- It unifies every position under the same mechanism — no fixed n-gram.
- The prompt's influence on generation flows through self-attention layers.
- Cross-attention (decoder reads encoder) is the same machinery with a second sequence.
Mental model
Everyone re-reads the room's notes simultaneously
Self-attention
Each attendee tests their question (query) against everyone else's index card (keys) and writes their own card (value) mixed with the most relevant ones. The room's summary changes after each pass — exactly what layered self-attention does to tokens.
How it works
- Input tokens are all treated alike — same projection stack computes Q, K, V.
- Scores between every pair fill the attention matrix (N×N).
- The mask (causal) zeroes future positions for language models.
- Weighted values sum into each token's new hidden state.
- A residual + normalization + MLP finishes each transformer block.
Self-attention rewrites each token.
In production
- Every LLM decoder step is causal self-attention over the prompt-plus-so-far.
- Retrieval systems emulate attention: query tokens attend a fetched document's tokens.
- Compression/summarization uses self-attention across the doc to find what matters.
Best practices
- Treat self-attention layers as parallel 'readers' — head count is capacity, not truth.
- Watch the attention matrix for collapsed/rank-deficient patterns in debugging.
- Leverage fused kernels (FlashAttention) — pure self-attention is IO-bound.
Common mistakes
- Forgetting the causal mask in decoder-only models (future leakage).
- Assuming prompt order matters little — it does; order IS content to the model.
- Measuring 'attention strength' without accounting for mask/scale.
Common issues
Model gives short shrift to early contextCausal/future masking or long-range dilution
Rephrase priming near the end; use retrieval to resurface it at the end.
Very slow on 50k-token promptsO(N^2) self-attention cost
Use efficient attention variants or reduce effective context.
Debugging tips
- Head attention matrix all-zero rows → dead heads, worth inspecting via logits.
- Generation ignores part of the prompt → inspect attention spread across prompt positions.
- Speed drop on long prompts → quadratic attention cost dominates; trim or window.
Interview checkpoint
Hands-on lab
- Build Q, K, V from a toy token matrix
- Compute scaled dot-product scores and softmax weights
- Verify that attention is a weighted sum of V
Scaled Dot-Product Attention by Hand
30 min · intermediate
Quick quiz
Self-attention queries, keys and values come from…
Order of tokens never matters to self-attention.