Tokens Are Not Words: What Every Developer Building on LLMs Actually Needs to Know

You’ve been paying for tokens since the day you made your first API call. You know the rough “4 characters per token” heuristic. You’ve probably hit a context limit at least once and scrambled to truncate your prompt.

But if someone asked you why “strawberry” costs 1 token but “Strawberry” might cost 2 depending on the model, or why the same sentence in Arabic costs three times more than in English, or why OpenAI’s tokenizer and Claude’s tokenizer will produce different token counts for identical input — could you explain it?

This isn’t academic trivia. It has direct consequences for your API bills, your context window math, your RAG chunk sizing, and some subtle model behavior bugs you probably haven’t diagnosed yet. Let’s go deep.

The Core Problem: LLMs Don’t Read Text

A transformer model operates on integer IDs. Not characters, not words — integers. The tokenizer’s entire job is to convert raw text into a sequence of these integers before the model sees anything, and then convert the output integer sequence back into text afterward.

The question every tokenizer has to answer: how do you turn an open-ended, unbounded human vocabulary into a fixed-size set of integer IDs?

Three approaches dominate the industry:

Word-level tokenization — every word is its own token. Clean, intuitive, and catastrophically broken in practice. You’d need a vocabulary of millions to cover all words in all languages, and you still can’t handle “unfriendable” or “bitcoiners” or any neologism.
Character-level tokenization — every character is a token. Vocabulary shrinks to dozens. But now the sequence length explodes: “tokenization” becomes 13 tokens instead of 1. The model has to learn relationships over absurdly long sequences.
Subword tokenization — the pragmatic middle ground that every major LLM uses. Common words become single tokens. Rare words get split into recognizable pieces. The vocabulary stays manageable.

Subword tokenization is the game, and Byte Pair Encoding (BPE) is how most of the industry plays it.

How BPE Actually Works

BPE was originally a data compression algorithm from 1994. It was adapted for NLP in 2015 and then popularized for LLMs by OpenAI’s GPT-2 paper. The core idea is elegant: start with the smallest possible units, then iteratively merge the most common pairs until you hit a target vocabulary size.

Here’s the training process in concrete terms:

Step 1: Initialize with bytes. Modern BPE doesn’t start with characters — it starts with UTF-8 bytes. That gives you a base vocabulary of exactly 256 tokens (the full byte range), which guarantees you can represent any input without an “unknown token” fallback. This variant is called Byte-Level BPE, and it’s what GPT-2 introduced and every major model since has used.
Step 2: Count pair frequencies. Scan the entire training corpus and count every adjacent pair of tokens. At initialization, those pairs are byte pairs. “the” would generate pairs (t,h) and (h,e).
Step 3: Merge the most frequent pair. Take the pair with the highest frequency across the corpus and merge it into a new single token. Now (t,h) might become the token th with some assigned ID, say 257.
Step 4: Repeat. After each merge, the corpus is updated with the new token, new frequencies are counted, and the next most common pair gets merged. This continues until you’ve hit your target vocabulary size.

What you end up with is a vocabulary where common English words like “the”, “and”, “is” are single tokens, moderately common subwords like “ing”, “tion”, “er” are single tokens, and rare or constructed words get assembled from whatever pieces exist.

The vocabulary size is a hyperparameter with real tradeoffs. GPT-2 used 50,257 tokens. GPT-4 scaled to 100,256 (the cl100k_base encoding). GPT-4o doubled that to approximately 200,000 (o200k_base). Claude uses a vocabulary in a similar range. The general trend across the industry is toward larger vocabularies because more tokens means shorter sequences means cheaper inference.

But there’s a cost: a 200K vocabulary means the embedding matrix alone — before you even get to the attention layers — consumes 200,000 × (embedding dimension) parameters. At an embedding dimension of 4096, that’s 800M parameters just for the lookup table.

The Regex Layer Nobody Talks About

Here’s something most tokenizer explanations skip: raw BPE would happily create tokens that span word boundaries. A token like “ the” (space + “the”) and “the” would be separate entries, and BPE would potentially create tokens that fuse unrelated words together based purely on frequency — which would be semantically incoherent.

The fix is pre-tokenization with regex. Before BPE runs at all, the text is split by a regex pattern that defines category boundaries. The pattern ensures BPE merges never cross word boundaries, never fuse a letter with a digit, and handles contractions like “don’t” as separate units.

GPT-4’s regex pattern (cl100k_base) looks like this:

'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+

That pattern splits on English contractions, limits number runs to groups of 3 digits (so “1234” becomes [”123”, “4”] rather than one token), and handles Unicode letters with \\p{L}. The {1,3} limit on numbers is precisely why LLMs have historically been terrible at arithmetic — they literally never see full multi-digit numbers as atomic units.

GPT-4o’s o200k_base updated this regex significantly to handle non-Latin scripts more efficiently, which is one of the reasons it costs fewer tokens for multilingual content than its predecessor.

The Three Tokenization Algorithms Used in Production

BPE + tiktoken (OpenAI, Llama 3+, Mistral)

tiktoken is OpenAI’s open-source tokenizer library, written in Rust with Python bindings. It’s 3–6x faster than comparable implementations. The library handles the cl100k_base encoding (GPT-3.5, GPT-4), the p50k_base encoding (older Codex models), and the o200k_base encoding (GPT-4o).

One important implementation detail in tiktoken: when a complete token already exists in the vocabulary as a predefined entry, it bypasses the BPE merge rules and returns directly as a single token. “hugging” might be in the vocabulary directly, so it returns as one token even if the standard merge chain wouldn’t produce it. This increases efficiency but introduces subtle inconsistencies if you’re trying to predict tokenization behavior programmatically.

Llama 3 and Mistral adopted tiktoken (or tiktoken-compatible tokenizers) rather than building their own, which is why token counts from those models map closely to OpenAI’s.

SentencePiece (Gemini, Llama 2, T5, many multilingual models)

SentencePiece, developed by Google, takes a fundamentally different approach to pre-tokenization: it doesn’t do whitespace-based pre-splitting at all. Instead, it treats the entire text as a raw character stream, encoding spaces as a special character (▁, Unicode U+2581). This means “▁Hello” and “Hello” are distinct in the vocabulary, and the model explicitly learns where word boundaries occur rather than having them imposed by regex.

SentencePiece supports two subword algorithms internally: BPE and Unigram Language Model. The Unigram approach works in reverse compared to BPE — instead of starting small and merging up, it initializes with a large candidate vocabulary and iteratively removes the tokens whose removal causes the least increase in encoding loss. The result is a probabilistic tokenizer where multiple valid segmentations exist for any input, and the highest-probability one is selected.

Unigram recovers linguistic morphology better than BPE in controlled experiments — it correctly splits “destabilizing” as “de” + “stabiliz” + “ing” rather than BPE’s “dest” + “abil” + “izing”. Whether that matters for model quality at scale is still contested, but it’s not zero.

Gemini uses SentencePiece. Gemma uses SentencePiece. Many multilingual-first models use SentencePiece because the no-whitespace-assumption design makes it genuinely language-agnostic — it handles Chinese, Japanese, and Thai without any language-specific preprocessing.

Claude’s Tokenizer (Anthropic)

Anthropic does not publish the full specification of Claude’s tokenizer, but it uses a vocabulary format compatible with the HuggingFace tokenizers library (the same JSON schema). It’s a BPE-based tokenizer, but trained on Anthropic’s own corpus rather than being derived from tiktoken.

The practical consequence: you cannot use tiktoken to get accurate token counts for Claude. You need either the Anthropic SDK’s client.messages.countTokens() endpoint, or a local approximation using the HuggingFace tokenizer utilities. The counts will be close to GPT-4 estimates but not identical — they’ll diverge more on code, non-English text, and specialized vocabulary.

Why the Same Text Costs Different Tokens on Different Models

This is where the business implications get real. Consider the Arabic sentence:

الذكاء الاصطناعي يغير العالم

On GPT-2’s r50k_base tokenizer (vocabulary ~50K), this produces approximately 21 tokens. On GPT-4o’s o200k_base, the same sentence produces around 5–7 tokens. The text is identical. The cost difference is 3–4x purely from the tokenizer choice.

This isn’t GPT-4o being “smarter about Arabic” at inference time. It’s that the o200k_base vocabulary was trained on a much larger and more diverse multilingual corpus, so more Arabic subwords made it into the vocabulary as single tokens rather than being assembled byte-by-byte.

The same pattern applies to code. GPT-4’s vocabulary was trained on a large volume of source code (eyeballing the token list, roughly half the vocabulary is code-related), so tokens like ValidateAntiForgeryToken or _InternalArray appear as single entries. Models trained primarily on natural language will tokenize the same code snippets into many more tokens.

The practical impact for developers:

Multilingual apps: The same semantic content costs dramatically different amounts per model. If you’re processing Arabic, Chinese, or Korean at scale, the tokenizer efficiency gap between models is a real line item in your infrastructure budget.
Code-heavy RAG: If your retrieval system chunks source code by token count, the chunk sizes will be inconsistent across models. 512 tokens of Python in one model might be 700 tokens in another.
Prompt optimization: Techniques that work on GPT-4 may need re-tuning on Claude or Gemini because the token boundaries hit differently. A carefully token-budgeted prompt doesn’t transfer between model families.

The Pathologies: What Tokenization Breaks

Arithmetic

GPT-3.5 and GPT-4 tokenize multi-digit numbers by capping digit runs at 3 characters ({1,3} in the regex). “1234” becomes [”123”, “4”]. “5678” becomes [”567”, “8”]. Now add those two numbers and the model needs to add misaligned columns — “4” + “8” doesn’t align to produce the carry correctly because the token boundaries don’t match.

A 2024 ICLR paper (Singh and Strouse) showed that right-to-left tokenization of numbers improves arithmetic accuracy by over 22 percentage points because it aligns the least-significant digits. The model architecture hasn’t changed — just the token boundary convention.

Spelling and character counting

“Strawberry” might be a single token (integer ID X). When you ask the model how many R’s are in “Strawberry”, it’s being asked to decompose token X into its constituent characters. The model doesn’t inherently see the letters — it sees the integer. It has to have learned, from training data, that token X corresponds to the sequence S-t-r-a-w-b-e-r-r-y. If that association wasn’t reinforced enough during training, it fails.

This is why early GPT-4 consistently said “Strawberry” has two R’s — it had to reconstruct character-level information from token-level embeddings, and got it wrong.

Glitch tokens

Some tokens exist in the vocabulary that were present in the training data in pathological contexts — Reddit usernames, server identifiers, garbage text scraped from obscure corners of the web. These tokens have poorly formed embeddings because they appeared rarely and in incoherent contexts. Asking the model to repeat or engage with these tokens produces unpredictable behavior. This is the mechanism behind “glitch tokens” that caused early jailbreaks and adversarial prompts.

Token boundary effects in generation

When a model generates text, it outputs one token at a time. The choice of token at position N influences the probability distribution at position N+1. If a common phrase like “don’t” exists as a single token but “do not” is two tokens, the model’s completion behavior can differ subtly depending on which representation appears in the context. Most of the time this doesn’t matter. In edge cases where the model is near a decision boundary, it can produce different outputs for semantically identical prompts written with different tokenization profiles.

What This Actually Means for Your API Bills

Token pricing is asymmetric across providers: output tokens are priced at 2–5x the rate of input tokens because generation is computationally more expensive than encoding. Both costs are directly determined by token count, which is directly determined by tokenizer efficiency.

The same 1,000-word English prompt runs to approximately 1,300 tokens on GPT-4o’s o200k_base. With a smaller vocabulary tokenizer, the same text might produce 1,500+ tokens. That gap compounds if you’re running millions of requests.

For non-English text the gap is larger. For code, it depends entirely on what’s in the vocabulary. For structured data like JSON, the tokenization is often surprisingly expensive — each curly brace, colon, and quote may be its own token.

Practical things to do with this knowledge:

Count tokens before you send. Use tiktoken for OpenAI models, the Anthropic SDK’s countTokens endpoint for Claude, and model-specific tools for everything else. The “4 characters = 1 token” heuristic is too inaccurate for production budget planning.
Measure tokenization efficiency on your actual data. Generic benchmarks don’t reflect your use case. If you’re running a code-heavy assistant and considering switching models, measure token counts on a representative sample of your actual prompts before projecting costs.
Be skeptical of “context window” comparisons. 128K tokens on GPT-4 and 200K tokens on Claude are not the same effective context depth if the tokenizers are meaningfully different in efficiency for your content type. A model with 200K tokens and a less efficient tokenizer might give you less actual text than a 128K model with a more efficient one.
Design RAG chunks around tokens, not characters. Most RAG chunk size guidance says things like “split at 1000 characters”. That’s cargo cult advice. You want semantically coherent chunks that fit within a consistent token budget. Measure the token count of your chunks, not the character count.

The Horizon: Tokenization-Free Models

The field is actively exploring models that bypass tokenization entirely by operating on raw bytes or characters. ByT5 demonstrated in 2022 that transformers processing raw bytes can be competitive with token-level models. The appeal is obvious: all the pathologies described above — arithmetic failures, spelling errors, multilingual cost inequality, glitch tokens — disappear if there’s no tokenization layer.

The problem is compute. Byte-level sequences are 3–4x longer than token sequences for the same text, and attention complexity is quadratic in sequence length. A 128K token context window becomes a 384K–512K byte context window. Current hardware makes that prohibitively expensive at scale, but as the efficiency of attention mechanisms improves (sparse attention, linear attention variants, hardware advances), byte-level models become more viable.

Whether the next generation of frontier models will have tokenizers at all is a genuinely open question.

The Summary

Tokenization is the layer between human language and the mathematics of the transformer. The algorithm (BPE for most models), the vocabulary size, the training corpus, and the pre-tokenization regex together determine:

How many tokens your prompt costs
How efficiently non-English content is encoded
Whether the model can perform arithmetic
How the model represents rare or compound words
How costs compare across providers for your specific workload

When you switch model providers, you are switching tokenizers. When you compare context windows across providers, you are comparing tokenizer-relative capacities. When you size RAG chunks or budget prompt length, you are doing tokenization accounting.

The “~4 characters per token” rule of thumb was never meant to be your production budget model. Now you know enough to do it properly.

Under The Hood covers the mechanics that the AI product ecosystem hand-waves. If this was useful, share it with the engineer on your team who still thinks tokens and words are the same thing.