Tokenization
Intermediate 11 min read Tokenization

Special Tokens

The punctuation of the vocabulary

Special tokens are reserved vocabulary entries with syntax power: end-of-sequence, padding, role markers, tools and JSON boundaries. Misusing them silently breaks fine-tunes and chats.

special tokensEOSBOSsystemtool tokensreserved vocabulary
Prerequisites: Tokenization Basics
Before we start

What you will learn

  • Identify the core special tokens
  • Explain their roles in chat and training
  • Avoid EDGE cases that corrupt them
Introduction

What is this?

Beyond subword pieces, vocabularies reserve special tokens — end-of-sequence (EOS), begin/end markers, role delimiters (system/user/assistant), tool-call tokens and JSON boundaries. Generation ends at EOS; chat formats mark roles with them; structured output models leg them into JSON. They are first-class signals, not ordinary text.

Motivation

Why do you need it?

  • Generation relies on EOS to stop — misconfiguring it produces infinite runs.
  • Chat/tool templates depend on exact role tokens; drift breaks parsing.
  • Structured-format models output JSON through reserved delimiters.
  • Fine-tune data built with wrong specials creates irreproducible checkpoints.
Think about it

Mental model

Imagine

Punctuation in a transcript

Think of

Special tokens

A period stops the sentence; quote marks fence quoted speech; '[end of tape]' tells the transcriber to stop. If the transcriber confuses quotes with narration, the whole interview is misread. Special tokens do exactly this structural work.

Mechanics

How it works

  • The vocab reserves dedicated IDs (e.g. <|endoftext|>, <|system|>, <|assistant|>).
  • Chat protocols interleave reserved role markers around user/assistant content.
  • Tool schemes add <|tool_call|>/<|tool_response|> openings.
  • Structured-output checkpoints emit JSON between boundary tokens for reliable extraction.
  • LLM training maximizes EOS probability at stop points — conditioning you must preserve.

Special tokens orchestrating a chat.

BOS
<|system|>
<|user|>
<|assistant|>
EOS
Real world

In production

  • Trust the SDK to place role tokens; never hand-roll templates without review.
  • Tool-calling parsers must tolerate escaped boundary tokens inside tool args.
  • Fine-tune data must follow the exact role-special scheme of the target checkpoint.
Guidelines

Best practices

  • Use the provider's message format — do not re-invent role markers.
  • Validate that EOS actually terminates; cap max_tokens as a safety net.
  • Sanitize user content so it cannot forge special tokens.
Watch out

Common mistakes

  • Stripping or renaming specials and 'fixing' the template yourself.
  • Letting user text smuggle <|assistant|>-like tokens (format injection).
  • Training on data whose roles don't match the deployed template.
Root cause analysis

Common issues

Error
Generation loops forever
Root cause

EOS not recognized in stop-sequence config

Fix

Set the correct stop tokens; raise max_tokens guard.

Error
Tool parser mis-splits on boundary tokens in args
Root cause

Escaping not applied to embedded specials

Fix

Escape or encode the argument payload before template insertion.

When things break

Debugging tips

  • Runs never end → EOS absent or masked; check stop-sequence config.
  • Assistant parsers see user-tagged blocks → role tokens misaligned or escaped wrong.
  • Fine-tuned model stops immediately → your data ended everything with EOS rather than the task answer.
Interviews

Interview checkpoint

Practice

Hands-on lab

Token-Count Every Major Slot
30 min · beginner
In progress
Objectives
  • Count tokens for a realistic chat multi-turn prompt
  • Verify rule-of-thumb estimates against a real tokenizer
  • Fix a budget blowout by trimming the largest slot

Token-Count Every Major Slot

30 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which special token's presence normally ends generation?

2

User text containing role-like tokens is harmless inside a chat template.

Go deeper

Official documentation

Special tokens (Hugging Face)
docs