Tokenization Basics
How text becomes numbers
Models read tokens, not words or characters. Tokenization chops text into a vocabulary of subword units — a process with its own costs, quirks and failure modes.
What you will learn
- Explain what a token is and why we need them
- Measure token counts of real text
- Spot when tokenization fights your data
What is this?
Tokenization is the lossy split of text into a fixed vocabulary of subword units (tokens) mapped to integer IDs. The model predicts vocab IDs; generation starts from IDs. Token choice shapes cost, quality and edge behavior dramatically — one 'token' is not one word, and rules differ per script and language.
Why do you need it?
- Billing and latency count tokens, not characters — get the arithmetic right.
- Vocabulary coverage decides how well rare names, code and non-Latin scripts perform.
- Token counters and SDK checks can disagree; knowing basic tokenization avoids surprises.
- Pathological tokenization (spaces, emojis, code) quietly embarrasses products at the edge.
Mental model
Splitting a novel into note cards
Tokenization
You'd cut the book into cards of a few words each (not one per letter, not one per chapter) so a reader can arrange them in any order. The card size (token size) trades vocabulary memory against flexibility; some sequences just get awkward cuts.
How it works
- A precomputed vocabulary maps token strings ↔ integer IDs.
- A tokenizer walks text, greedily matching the longest vocabulary pieces.
- Common words cost 1 token; rare/uncommon terms often split into several.
- The model consumes ID sequences; generated IDs get detokenized back to text.
- Whitespace and punctuation usually become their own tokens.
Text → IDs → model → IDs → text.
Examples
Estimate tokens without an API
import re def rough_tokens(text): # a very rough approximation: 4 chars per token for English return max(1, (len(text) + 3) // 4) s = "The quick brown fox jumps."print(len(s), "chars ~", rough_tokens(s), "tokens") # real rule of thumb: English ≈ 4 chars/token, code ≈ 3.5, few languages differprint("note: 1 token ≈ 4 chars in English")Rough but reliable for budgeting; exact counts need the provider's tokenizer.
In production
- Cost models multiply token counts from prompt + completion.
- Context budgeting uses actual tokenized lengths, not characters.
- Logging token counts per call drives spend dashboards.
Best practices
- Measure real token counts (provider tokenizers) instead of character math.
- Design prompts to reuse frequent vocabulary (tight, common phrasing).
- Watch token counts across locales — non-Latin text costs more tokens.
Common mistakes
- Assuming token ≈ word for billing.
- Counting context in characters and overflowing windows.
- Ignoring that whitespace/punctuation inflate counts.
Common issues
Prompt overflows '9k chars' but token limit hitsChars-to-tokens ratio underestimated
Tokenize before the call and log real counts.
Generated output loses leading spacesDetokenization trims undocumented whitespace tokens
Handle code formatting with explicit markers or structured outputs.
Debugging tips
- A 1000-char Japanese string may be 2x the tokens of English.
- Trailing spaces or leading whitespace in prompts add surprising tokens.
- Emoji sequences tokenize into several units — often costly.
Interview checkpoint
Hands-on lab
- Compare characters, words and estimated tokens on the same string
- Apply a length-based token estimate and reason about its error
- Budget prompts from counts without a live tokenizer
Estimate Token Counts Like a Model
25 min · beginner
Quick quiz
The model's basic unit of processing is a…
Tokenization is lossless: any byte sequence maps to exactly one token run.