Transformers
Intermediate 12 min read Transformers

Causal Masking

No looking ahead

Decoder models must never peek at the future: a causal mask zeroes attention to later positions so each token sees only itself and its predecessors. It's the grammar license of next-token training and streaming generation.

causal maskfuture tokensmasked attentionstreamingautoregressive
Prerequisites: Multi-Head Attention
Before we start

What you will learn

  • Explain the causal mask's geometry
  • Implement it in a few lines
  • Connect it to training correctness and streaming
Introduction

What is this?

A causal mask sets attention scores from token i to all tokens j > i to −∞ (softer weights go to zero), making every position attend only to the past and itself. This makes the model's distribution decomposable — P(token sequence) = Π P(token_i | past) — which is exactly what legitimate next-token training and streaming generation require.

Motivation

Why do you need it?

  • Training on future-peeking models silently overfits to lookahead and misgeneralizes.
  • The chain-rule factorization depends on it.
  • Streaming decode at inference follows the same causal order.
  • Leaking the mask (even in fine-tuning) corrupts quality everywhere.
Think about it

Mental model

Imagine

A one-way street in class

Think of

The causal mask

You may read everything that came before the current note but nothing after — notes must be written left to right. A student who cheats by reading the answer section learns the test, not the subject; that's exactly what a lookahead 'training leak' does to a model.

Mechanics

How it works

  • Attention score matrix is N×N; for every (i, j) with j > i, set score to −∞.
  • Softmax turns −∞ into ~0 weight; the future may not influence the present.
  • Each token's output depends only on ≤-itself tokens.
  • At inference, decode tokens are appended as they're generated — same causal rule.
  • Encoder models omit it (bidirectional); decoder models require it.

The mask is a lower triangle.

Tokens 0..N
Fill
Softmax
Decode
Hands-on

Examples

Causal mask in numpy

Causal mask in numpy
import numpy as np
 
n = 5
scores = np.ones((n, n)) * 0.1
mask = np.tril(np.ones((n, n)), k=0)          # lower triangle
scores = np.where(mask == 1, scores, -np.inf) # future -> -inf
weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights = weights / weights.sum(axis=1, keepdims=True)
print(np.round(weights, 2))

Row i has mass only on columns 0..i — the model can't see beyond itself.

Real world

In production

  • Every decoder training step embeds a causal mask; removing it quietly breaks training.
  • Speculative decoding uses the same causal view for draft and verify.
  • Streaming a long answer is literally walking the mask's diagonal.
Guidelines

Best practices

  • Use framework causal options; never hand-roll unless profiling says otherwise.
  • Verify the mask when re-using position offsets larger than zero.
  • Guard fine-tune contracts against masked-attention regressions.
Watch out

Common mistakes

  • Hand-implementing attention without the mask — the model sees the future and looks great, then falls apart.
  • Applying causal masks in encoder-like passages where bidirectional reading is wanted.
  • Asymmetric masks in batched decoding → cache/order bugs.
Root cause analysis

Common issues

Error
Very low train loss, clearly bad generated text
Root cause

Future leakage — mask missing or wrong

Fix

Confirm the triangular mask and retrain/retune; drop leaky checkpoint.

Error
Decode duplicates tokens during streaming
Root cause

KV cache and mask mis-indexed during append

Fix

Assert cache length == position; rebuild cache on jump.

When things break

Debugging tips

  • Suspiciously high train loss reduction with poor test → likely lookahead leak across accidental positions.
  • Streaming jumps ahead (generates the same words twice) → mask/cache indices out of sync.
  • Strange behavior on position-shifted prompts → verify mask orientation with the position offset.
Interviews

Interview checkpoint

Practice

Hands-on lab

Implement Causal Attention from Scratch
40 min · intermediate
In progress
Objectives
  • Compute self-attention with and without a causal mask
  • Verify the mask zeroes future influence
  • Confirm masked output equals the equivalent causal-slice run

Implement Causal Attention from Scratch

40 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A causal mask guarantees each token attends only…

2

Encoder-style models usually use causal masks.

Go deeper

Official documentation

Attention masks (Hugging Face)
docs