How LLM Tokenizers Actually Work
BPE, o200k_base, cl100k_base — what actually happens when text becomes tokens, and why 1 token is not 1 word.
Every prompt you send to an LLM is first split into tokens — chunks of text that the model processes individually. Understanding tokenization is the key to estimating costs, fitting context windows, and predicting where the model will struggle.
Byte Pair Encoding (BPE)
Modern tokenizers use BPE, a compression algorithm. It starts with every byte as its own token, then repeatedly merges the most common adjacent pair into a new token until the vocabulary reaches a target size.
"tokenization" → ["token", "ization"]
"unhappiness" → ["un", "happiness"]
"rustacean" → ["rust", "acean"] ← rare word, more fragments
This is why common words are single tokens and rare words split into fragments — the vocabulary is optimized for frequency.
The OpenAI encodings
| Encoding | Vocabulary | Used by |
|---|---|---|
cl100k_base | ~100k | GPT-3.5, GPT-4, text-embedding models |
o200k_base | ~200k | GPT-4o, GPT-4o-mini, o1, o3 |
p50k_base | ~50k | Codex (deprecated) |
The jump from 100k to 200k vocabulary means GPT-4o can represent more words as single tokens, especially in non-English languages. A Portuguese sentence that took 40 tokens with cl100k_base might take 28 with o200k_base.
Rules of thumb
- 1 token ≈ 4 characters of English text, or roughly 0.75 words.
- Code and JSON tokenize denser — expect more tokens per character because brackets, quotes, and indentation each cost tokens.
- Non-English text costs 2-4x more tokens per word because the vocabulary is English-heavy.
- Leading whitespace and capitalization matter:
Helloandhelloare different tokens.
Why this matters for cost
Providers charge per million tokens — separately for input and output. Output is typically 3-5x more expensive:
| Model | Input / 1M | Output / 1M |
|---|---|---|
| GPT-4o | $2.50 | $10.00 |
| Claude 3.5 Sonnet | $3.00 | $15.00 |
| Gemini 1.5 Flash | $0.075 | $0.30 |
A prompt that's 500 tokens over the optimal length, sent 10,000 times a day, costs $12.50/day extra on GPT-4o — just for input.
Use the Token Counter to measure exact token counts across 100+ models and compare per-million-token pricing side by side.