Transformers
Intermediate 17 min read Transformers

Transformer Architecture

The block structure behind every modern LLM

A transformer stacks attention and feedforward blocks with residual adds and normalization, in encoder, decoder, or encoder-decoder arrangements. "Attention is all you need" — plus linear layers, LayerNorm, residuals and position injectors.

transformerencoderdecoderresiduallayer normFFNattention block
Prerequisites: Self-Attention & Multi-Head Positional Encoding
Before we start

What you will learn

  • Map the encoder/decoder arrangement and its three attention types
  • Explain residual + LayerNorm in each sub-layer
  • Describe the FFN and what it adds to attention
Introduction

What is this?

A transformer block = multi-head self-attention + a position-wise feed-forward network (two linear layers with GELU/ReLU), each wrapped in residual connections and LayerNorm. Stacks of these form an encoder (bidirectional), a decoder (causal, with cross-attention to the encoder), or an encoder-decoder translation model. It has no recurrence at all — position comes from embeddings, relationships come from attention.

Motivation

Why do you need it?

  • Full parallelism made large-scale NLP training possible.
  • Residuals let the stack stay trainable hundreds of layers deep.
  • Causal masking in decoders gives autoregressive generation.
  • The same block transfers to vision, audio and multimodal via adapted inputs.
Think about it

Mental model

Imagine

An analytical council with specialists and a scribe

Think of

Transformer blocks

Each round the council (self-attention) has everyone read everyone's brief — while specialist clerks (FFN) then think over each person's specific concern. After each session, each member adds their own notes to what they already knew (residual) and renormalizes (LayerNorm), staying coherent no matter how many sessions pass.

Mechanics

How it works

  • Sub-layer 1: x + LayerNorm(MultiHeadSelfAttn(x)) — the residual add keeps gradient paths clear.
  • Sub-layer 2: x + LayerNorm(FFN(x)) — FFN = Linear(d→4d) → GELU/ReLU → Linear(4d→d), per-position.
  • Encoder = N identical blocks using bidirectional self-attention.
  • Decoder = N blocks of causal self-attention + cross-attention over encoder outputs + FFN.
  • Output head maps the last hidden state to next-token logits (tied to embedding matrix often).

Two sub-layers inside one transformer block.

Input x
Self-attention
Add & norm
FFN
Add & norm
Real world

In production

  • Encoder-only: BERT-style for classification, retrieval, embeddings.
  • Decoder-only: GPT-style autoregressive generation — the default LLM architecture.
  • Encoder-decoder: translation/summarization (BART/T5) where cross-attention shines.
Guidelines

Best practices

  • Scale depth and width with data and compute budget — the training-time laws matter.
  • Keep embedding/width multiples of head count (head dim ≈ 64).
  • Use causal masks for any decoder and an attention mask for padding.
  • Apply LayerNorm pre-attention (pre-norm) for stability with deep stacks.
Watch out

Common mistakes

  • Mixing positional injection AFTER attention (position must ride the token embeddings into attention).
  • Reading the FFN as memory — it's a per-position nonlinear transform, not an attention head.
  • Training a full-decoder causal stack on bidirectional tasks and wondering why 'MASK' can't see the answer.
  • Skipping residuals at a depth where gradients then vanish.
Root cause analysis

Common issues

Error
Deep stack training diverges at warm-up
Root cause

Post-norm + poor init compounding — early attention logits explode

Fix

Switch to pre-norm (LayerNorm before attention/FFN), scale init, and long warm-up.

Error
Decoder attends to future during evaluation
Root cause

Mask not applied at inference rollout

Fix

Always build the causal mask in the forward path, not just in the training graph.

Error
Embedding dimension not divisible by heads
Root cause

Head split attempts produce ragged tensors

Fix

Keep width = n_heads × head_dim and set head_dim ≈ 64.

When things break

Debugging tips

  • Print a block's input vs output norms — stable stacks keep them in similar scale after norm.
  • Verify attention mask shapes (T, T) per head, and that the decoder never attends to its own future.
  • Run a tiny single-block model on a 4-token input; examine every intermediate shape.
  • Compare pre-norm vs post-norm stability on your longest sequence before tuning further.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The 'add' in each transformer sub-layer means:

2

The feed-forward network in a transformer mixes information across all tokens.

Go deeper

Official documentation

The Illustrated Transformer
docs