Tokenizers are the bridge between human-readable text and the numerical input that neural networks actually process. Every interaction with a language model — every prompt, every completion, every embedding — begins with tokenization: splitting raw text into discrete units called tokens and mapping each token to an integer ID from a fixed vocabulary.
Understanding tokenizers is essential for AI engineers because they directly affect cost, context window usage, and model behavior. A sentence that tokenizes into 50 tokens with one model might tokenize into 80 tokens with another, meaning different costs and different amounts of context budget consumed. The tokenizer determines how efficiently a model can represent your data, and mismatches between the tokenizer’s training corpus and your domain can create surprising inefficiencies.
Modern LLMs universally use subword tokenization — a middle ground between character-level tokenization (tiny vocabulary, long sequences) and word-level tokenization (huge vocabulary, unknown word problem). Subword tokenizers learn to split text into meaningful pieces: common words remain intact as single tokens, while rare or compound words decompose into familiar subwords. The word “unhappiness” might become ["un", "happiness"] or ["un", "happ", "iness"] depending on the tokenizer.
To understand why subword tokenization dominates, consider the alternatives:
[UNK] — a total information loss.[UNK] tokens.Subword tokenization learns a vocabulary from a training corpus by iteratively merging frequently co-occurring character sequences. The result: common words stay intact as single tokens ("the" → [1]), while rare words decompose into familiar subwords ("unforgettable" → ["un", "forget", "table"]). This eliminates the unknown word problem while keeping sequences short.
BPE is the most widely used subword algorithm, powering GPT-2, GPT-3, GPT-4, Llama, Gemma, and Qwen models. Originally a data compression technique, it was adapted for NLP by Sennrich et al. in 2016.
The BPE training algorithm:
# Simplified BPE training illustration
corpus = "low lower newest widest"
# Step 0: Character vocabulary
# {"l", "o", "w", "e", "r", "n", "s", "t", "i", "d", " "}
# Step 1: Most frequent pair is ("e", "s") → merge to "es"
# Step 2: Most frequent pair is ("es", "t") → merge to "est"
# Step 3: Most frequent pair is ("l", "o") → merge to "lo"
# Step 4: Most frequent pair is ("lo", "w") → merge to "low"
# ...continues until target vocabulary size reached
# At inference, BPE applies learned merges greedily:
# "lowest" → ["low", "est"] (both are known tokens)The key insight is that BPE merges are learned from data — the vocabulary naturally adapts to the language distribution. English text will develop tokens for common morphemes like "ing", "tion", "un", while Python code will develop tokens for "def ", "self.", "return ".
WordPiece is the tokenization algorithm behind BERT, DistilBERT, and Electra. It’s structurally similar to BPE but differs in merge selection. Instead of merging the most frequent pair, WordPiece merges the pair that maximizes the likelihood of the training corpus — essentially choosing merges that are statistically most informative.
BPE merges the pair with the highest frequency count. WordPiece merges the pair whose joint probability most exceeds the product of individual probabilities: score(a, b) = freq(ab) / (freq(a) × freq(b)). This means WordPiece prefers merging tokens that appear together far more than chance predicts, even if their absolute frequency is lower.
WordPiece uses the ## prefix to indicate continuation subwords. The word “unaffordable” might tokenize as ["un", "##afford", "##able"], where ## signals that the subword is not the start of a new word. This prefix convention helps the model distinguish word boundaries.
SentencePiece, developed by Google, is a tokenizer framework rather than a single algorithm. It can run BPE or Unigram tokenization, but its key innovation is language independence: it treats input as a raw byte stream without any language-specific preprocessing (no pre-tokenization into words, no whitespace rules).
This makes SentencePiece ideal for multilingual models because it handles scripts without whitespace word boundaries (Chinese, Japanese, Thai) just as naturally as space-delimited languages. SentencePiece powers T5, ALBERT, Llama, and many multilingual models.
# Using tiktoken (OpenAI's BPE tokenizer) for token counting
import tiktoken
# Get the tokenizer for GPT-4
enc = tiktoken.encoding_for_model("gpt-4")
text = "Tokenizers convert text to integer sequences."
tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Tokens: {tokens}")
print(f"Token count: {len(tokens)}")
print(f"Decoded tokens: {[enc.decode([t]) for t in tokens]}")
# Output:
# Text: Tokenizers convert text to integer sequences.
# Tokens: [3404, 12509, 5765, 1495, 311, 7698, 24630, 13]
# Token count: 8
# Decoded tokens: ['Token', 'izers', ' convert', ' text', ' to', ' integer', ' sequences', '.']The vocabulary size is a critical hyperparameter that trades off between compression efficiency and model size:
o200k_base encoding)count_tokens API endpoint)Larger vocabularies mean more common multi-character sequences get their own token IDs, producing shorter token sequences for the same text. This saves context window space and reduces inference cost. However, larger vocabularies also increase the embedding table size (each token needs a learned vector), which adds to model parameters and memory requirements.
Every tokenizer reserves special tokens for model-level signaling:
<|endoftext|> — End of sequence / document separator (GPT models)[CLS] / [SEP] — Classification token and separator (BERT)<s> / </s> — Beginning and end of sequence (Llama, T5)[PAD] — Padding token for batch processing<|im_start|> / <|im_end|> — Chat message delimiters (ChatML format)These tokens are crucial for multi-turn conversation formatting and are invisible to end users but determine how the model interprets prompt structure.
LLM API pricing is per-token. Before sending a request, you need to know how many tokens your prompt will consume. Each model family requires its specific tokenizer for accurate counting: tiktoken for OpenAI models, Anthropic’s count_tokens endpoint for Claude, and SentencePiece for Llama-family models. A rough heuristic is ~4 characters per token for English text, but this varies significantly across languages and domains.
When building RAG systems, you must fit retrieved chunks plus the system prompt plus the user query plus the expected output into the model’s context window. Tokenizing each component lets you compute exactly how much budget remains and whether you need to trim chunks. This is why “token budgeting” is a core context engineering skill — and it starts with accurate tokenization.
When chunking documents for embedding, your chunk size should be measured in tokens (not characters or words) because the embedding model processes tokens. A 512-token chunk might be 300 words or 2,000 characters depending on the content. Using the embedding model’s tokenizer to measure chunks ensures consistent semantic density across your vector store.
A critical pitfall: different models tokenize the same text differently. The sentence “I’m experiencing déjà vu” produces different token counts across GPT-4, Claude, and Llama because they use different vocabularies and merge rules. This matters when migrating between providers or running A/B tests — your cost projections and context budgets won’t transfer directly.
tiktoken for OpenAI, Anthropic’s API for Claude, sentencepiece for Llama/T5.