Transformers
Intermediate 13 min read Transformers

Multi-Head Attention

Many readers, one book

Multi-head attention runs several attention mechanisms in parallel with different projections, letting the model track pronouns, syntax and semantics at once — the reason a single softmax can't do everything.

multi-head attentionheadsconcatenationparallel attentionrepresentation
Prerequisites: Self-Attention
Before we start

What you will learn

  • Explain the head-split and merge
  • Appreciate head specialization
  • Tie head count to capacity and cost
Introduction

What is this?

Multi-head attention splits the model dimension into h parallel heads, each with its own Q/K/V projections. Each head runs self-attention independently; outputs concatenate and project back. Different heads learn different patterns — syntactic, relational, positional — giving one layer many 'reading strategies' instead of one average.

Motivation

Why do you need it?

  • A single attention bag averages patterns; heads can specialize while sharing capacity.
  • Heads are cheap extra capacity: the compute of h heads equals one big head, but with more general shapes.
  • Head ablation studies show some heads are near-irreplaceable (core expertise).
  • Increasing head count is a standard path to better representations.
Think about it

Mental model

Imagine

A relay team of readers

Think of

Multi-head attention

One reader catches pronouns, another notices dates, a third tracks who's doing what. They all read the same book (the sequence) and each contributes their finding; the note-taking merge produces the final understanding.

Mechanics

How it works

  • Input dimension d splits into h heads of d/h each via projections.
  • Each head computes its own attention output independently.
  • All head outputs concatenate back to d.
  • A final linear projection mixes across heads.
  • Models expose ~12..128 heads by size; capacity grows with h, cost grows linearly.

Split, attend, merge.

Input x
Head 1
Head 2..h
Concat
Project
Real world

In production

  • It's the default everywhere: all decoder blocks use multi-head attention.
  • Head pruning/ablation tweaks model quality/speed without retraining everything.
  • KV cache amortization naturally splits per head for serving speedups.
Guidelines

Best practices

  • Prefer framework defaults for head count; tune the model dimension, not h in isolation.
  • Expect heads to specialize; explore with visualizations when debugging behavior.
  • Watch memory: KV cache is per-head, so head count directly affects cache size.
Watch out

Common mistakes

  • Hand-splitting dimensions wrong (d/h not integer) breaks shapes.
  • Assuming every head needs identical behavior — specialization is the point.
  • Forgetting head dimension when budgeting KV cache memory.
Root cause analysis

Common issues

Error
Shape error after concat
Root cause

Head dimension mismatch

Fix

Verify d % h == 0 and correct split sizes.

Error
Memory growth with head count
Root cause

KV cache scales per head

Fix

Budget cache in d·layers·tokens; raise heads only with actual benefit.

When things break

Debugging tips

  • Odd head output → check d % h and projection shapes.
  • Inspect single-head attention matrices to spot collapsed leaders.
  • Memory blow-up with many heads + long context → cache size formula h·d/h... equals d — still watch count.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What gets concatenated at the end of multi-head attention?

2

Heads train to be identical redundant copies.

Go deeper

Official documentation

Transformer block visuals
docs