WordPiece & SentencePiece
Alternative tokenizers with different quirks
BPE isn't the only subword recipe. WordPiece (likelihood-maximizing, used by BERT) and SentencePiece (a trainable-text pre-splitter enabling multilingual tokenization) each carry different behaviors and trade-offs.
What you will learn
- Contrast WordPiece vs BPE objectives
- Explain SentencePiece's raw-text input
- Pick a tokenizer family by use case
What is this?
WordPiece builds vocabulary by maximizing training-data likelihood under a model of merges (used by BERT), BPE by pure frequency, and SentencePiece treats the raw byte stream directly — optional normalization, reversible, and friendly to multilingual/code data. Unigram (SentencePiece's other mode) iteratively drops the least useful pieces. Each bakes different trade-offs into the vocabulary.
Why do you need it?
- WordPiece's likelihood objective tends to yield 'less pure frequency' merges than BPE.
- SentencePiece enables vocabulary sharing across scripts and simplifies preprocessing pipelines.
- Unigram allows much larger candidate vocabularies and better compression of mixed-language data.
- Choosing matter because token vocab interacts with cost, quality and reuse when fine-tuning.
Mental model
Building blocks with different judging criteria
Tokenizer families
BPE welds the two most common blocks (frequency). WordPiece welds the pair whose addition most improves the model's fit (likelihood). SentencePiece sorts raw clay (bytes) before firing — and Unigram trims the least useful custom bricks from the kit.
How it works
- WordPiece: pick merges that maximize corpus likelihood under a probabilistic model; includes the distinctive ## prefix notation.
- SentencePiece: operates on raw text (no pre-split into whitespace words), trains from a token-character pool.
- Unigram: initialize a large candidate vocabulary, iteratively prune pieces that contribute least.
- All three emit subword integers consumed identically downstream.
- Normalization (NFKC etc.) config differs — the same model can tokenize differently per settings.
Three tokenizer families.
In production
- Multilingual serving pipelines commonly buy SentencePiece-based checkpoints.
- BERT-derived encoders keep WordPiece vocabularies you must match when fine-tuning.
- Mixed-language or code-heavy products favor SentencePiece/Unigram coverage.
Best practices
- Match tokenizer and checkpoint exactly — mismatches change token IDs.
- Pin tokenizer configuration (normalizer, vocab) in your repo for reproducibility.
- Test a multilingual sample through the tokenizer before committing to a checkpoint.
Common mistakes
- Mixing tokenizer versions between train and inference.
- Disabling SentencePiece normalization and breaking the pipeline's contract.
- Assuming 'the same vocab size' means similar coverage across trainer families.
Common issues
Checkpoint/tokenizer ID mismatchWrong vocab file or settings
Load the checkpoint's exact tokenizer; verify a known string's IDs.
Poor multilingual coverageVocab trained on narrow data
Choose SentencePiece/Unigram models trained on multilingual corpora or extend vocab.
Debugging tips
- Fine-tune produces garbling at the boundary — check tokenizer archetype matches checkpoint.
- Unknown pieces for your domain → the unigram/merge pool lacks them; check vocab coverage.
- NFKC-inconsistent counts across environments → pin normalization config.
Interview checkpoint
Quick quiz
Which tokenizer builds vocab by likelihood rather than raw frequency?
SentencePiece can tokenize without a pre-built whitespace word split.