Tokenization
Intermediate 13 min read Tokenization

Byte-Pair Encoding (BPE)

Merging your way to a vocabulary

BPE builds a vocabulary by repeatedly merging the most frequent pairs of characters/subwords until the budget runs out. It is the standard subword algorithm behind most LLMs.

BPEbyte-pair encodingmerge rulesvocabularysubword tokenizer
Prerequisites: Tokenization Basics
Before we start

What you will learn

  • Run BPE merges on a tiny corpus
  • Explain why BPE balances size and coverage
  • Relate BPE to token/cost behavior
Introduction

What is this?

Byte-pair encoding counts adjacent pairs in the corpus, merges the most frequent pair into one token, repeats until a vocabulary budget is reached. Common words become single tokens; rare ones stay fragmented. Its merge history IS the tokenizer — send text through the same greedy match.

Motivation

Why do you need it?

  • BPE is what nearly all LLM tokenizers actually do.
  • It explains tightly why frequent words cost one token and unusual ones many.
  • The merge training bakes in the corpus's language biases — visible as token asymmetry.
  • You can audit a BPE merge table to predict token behavior.
Think about it

Mental model

Imagine

Folding the folds of a shirt with a hot iron

Think of

BPE merges

The iron merges the two edges that touch most often into one crease, then presses again on the next most-common touching pair. After enough presses, the shirt has fewer, bigger panels — the most-used wrinkles became single pieces.

Mechanics

How it works

  • Start vocabulary = byte alphabet, track adjacent pairs and their corpus counts.
  • Merge the most frequent pair into a new token; counts update.
  • Repeat until the vocab budget is spent.
  • The final merge table is applied greedily at tokenize time.
  • Train Tokenizers to a budget 30k-100k+ depending on model/data.

The BPE loop.

Corpus
Count pairs
Merge top pair
Budget?
Final vocab
Hands-on

Examples

BPE on three words

BPE on three words
from collections import Counter
 
words = ["low", "lowest", "newer", "wider", "newer"]
vocab = set("".join(words))
pairs = Counter()
 
for w in words:
    seq = list(" " + w + " ")  # add word boundaries
    for p in zip(seq, seq[1:]):
        pairs[p] += 1
 
print(pairs.most_common(3))
print("merge candidates:", [a+b for (a,b), _ in pairs.most_common(3)])

The most frequent bigrams become new tokens: 'er', 'ew', 'low' typically win.

Real world

In production

  • Choosing an open checkpoint often means adopting its tokenizer — vocabulary is baked in.
  • Format-sensitive work (code, JSON) profits from checking the vocab has syntax tokens.
  • Domain jargon should appear in vocab; if not, expect fragmentation costs.
Guidelines

Best practices

  • Prefer official tokenizers over approximate libraries.
  • Test your top-100 natural strings for token counts.
  • When fine-tuning for a domain, consider a carefully extended vocabulary.
Watch out

Common mistakes

  • Extending a vocabulary without retraining merges — creates orphaned IDs.
  • Assuming a newer BPE is 'the same' across model versions.
  • Ignoring rare but expensive tokens (boilerplate strings, emoji).
Root cause analysis

Common issues

Error
Vocabulary too small for your domain
Root cause

Low-frequency domain subwords

Fix

Add domain text to merge training or pick a larger-vocab checkpoint.

Error
Inconsistent token counts across runtimes
Root cause

Mismatched tokenizer versions

Fix

Pin tokenizer/library versions and enforce exact counts in one source of truth.

When things break

Debugging tips

  • Odd split of UK-based spellings → check vocab language skew in trainset.
  • Code that 'eats' tokens: operators and symbols often force subword splits.
  • Some tokens break JSON strings when fused (e.g. '] }') — escape or format carefully.
Interviews

Interview checkpoint

Practice

Hands-on lab

A Tiny Byte-Pair Encoder
30 min · intermediate
In progress
Objectives
  • Implement pair counting and the most-frequent merge step
  • Train a few merge rules on a tiny corpus
  • Show that merges reduce the token count per sequence

A Tiny Byte-Pair Encoder

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

BPE merges the most frequent…

2

BPE vocabulary favors corpus-frequent subwords.

Go deeper

Official documentation

BPE walkthrough (HF course)
docs
tiktoken package
docs