Tokens and Tokenization
On this page
Your team ships a support bot. The bill for the first month is three times the estimate, users writing in Chinese hit the context limit long before English users, and one customer's 40-page contract makes the request fail outright. Every one of these surprises has the same cause: the estimate counted words or characters, and the model counts tokens.
This article explains what a token is, how text is cut into tokens, and why that one detail drives cost, context limits and speed in every LLM system you will build or be asked about.
How it works
Text in, numbers out
A language model does arithmetic on vectors; it cannot read letters. So before anything reaches the model, a tokenizer cuts the text into pieces from a fixed vocabulary and replaces each piece with its number in that vocabulary. The model reads a list of integers and writes a list of integers, and the tokenizer turns the output back into text.

Three things follow directly, and they are the reason interviewers ask about tokens at all:
- Price is per token, usually with a higher price for output tokens than input tokens.
- The context window (the most the model can see at once) is a token count, and it covers the prompt plus the answer.
- Speed is measured in tokens: time to the first token, then tokens per second.
Byte Pair Encoding, the common recipe
Most current LLM tokenizers use a variant of Byte Pair Encoding (BPE). It is learned once, from a large sample of text, before the model is trained:
- Start with the smallest units: single bytes (or characters), so any input can always be written down.
- Count every pair of neighbouring units in the training text.
- Merge the most frequent pair into a new unit and add it to the vocabulary.
- Repeat until the vocabulary reaches the target size, often somewhere between about 30,000 and 200,000 entries.

To encode new text, the tokenizer applies the learned merges in the same order. The result is a sensible compromise: common words like "the" or "cache" are a single token, a rare word such as a product name breaks into a few known pieces, and nothing is ever "unknown", because the tokenizer can always fall back to bytes.
Other schemes exist (WordPiece, Unigram as used in SentencePiece), but they share the idea: a fixed vocabulary of sub-word pieces, frequent things cheap, rare things split.
Why the same meaning costs different amounts
The vocabulary reflects the text it was learned from. If most of that text was English, English words got merged into long tokens, while other languages, code and unusual formatting stay in shorter pieces. So the same content can cost very different token counts.

A rough rule for English is about 4 characters, or three quarters of a word, per token. Treat it as a first guess only: it is wrong for other languages, for code, and across model families, because each family has its own tokenizer.
Trade-offs and costs
Vocabulary size. A bigger vocabulary means fewer tokens per sentence, so shorter sequences, cheaper attention and more text in the same window. The price is a larger embedding table and output layer, and each rare token gets fewer training examples. Recent models have moved toward larger vocabularies, partly to treat more languages fairly.
Sub-words versus characters. Character or byte-level models never split words oddly, but sequences become several times longer, and attention cost grows quickly with length. Sub-word tokenizers are the practical middle ground.
Side effects you can see. Because the model sees pieces, not letters, tasks such as counting letters in a word, reversing a string or exact arithmetic on long numbers are harder than they look. Numbers are often split into chunks of digits in ways that do not line up with place value.
In production
- Cost estimates. Budget from measured token counts on real traffic, split into input and output, since output tokens are usually priced higher. A system prompt of 2,000 tokens sent with every request is often the biggest line item.
- Context limits. Count tokens before you call the model. Reserve room for the answer: if the window is 128k tokens and you want up to 4k tokens back, the prompt must fit in about 124k.
- Latency. Each output token is one decoding step, so long answers are slow answers. Asking for a shorter format is often the cheapest speed-up.
- Failure modes. Truncating text by characters can cut a multi-byte character or a token in half; truncate by tokens instead. Changing models can change the tokenizer, so stored token counts, limits and cached prompts may all shift.
- Format choice. Compact formats (CSV, minified JSON, short keys) cut input tokens noticeably for structured data.
When it matters most
- Estimating cost or capacity for any LLM feature.
- Deciding how many retrieved chunks fit into a RAG prompt.
- Serving many languages: budgets and limits must be set per language, not globally.
- Debugging odd model behaviour around spelling, counting, numbers or rare names.
Example
A support bot sends a 1,500-token system prompt, about 3,000 tokens of retrieved help-centre text and the user's message (say 100 tokens), and gets back about 300 tokens. Assume a price of $1 per million input tokens and $4 per million output tokens (example prices only).
- Input: 4,600 tokens × $1 / 1M = $0.0046
- Output: 300 tokens × $4 / 1M = $0.0012
- About $0.006 per request, so roughly $6,000 per million requests.
What this tells you: the fixed system prompt and retrieved text are 97% of the input. Trimming the system prompt or retrieving fewer, better chunks saves more than shortening answers. If half the users write in a language that needs 1.5 times as many tokens, the real bill is noticeably higher than the English-only estimate.
Common mistakes
- Counting words or characters instead of tokens when estimating cost or checking limits.
- Forgetting the output when checking the context window, so long answers get cut off.
- Assuming one tokenizer for all models. Counts differ between model families; measure with the right one.
- Truncating by characters, which can split characters and tokens and corrupt the input.
- Blaming the model's reasoning for spelling or digit mistakes that come from tokenization.
Say this in the interview
"The model works on tokens, not words. Most tokenizers are BPE-style: a learned vocabulary of sub-word pieces, so common words are one token and rare ones split. That matters because price, context limit and latency are all counted in tokens, and the ratio changes with language and format. So I'd measure token counts on real traffic, budget input and output separately, and keep room in the window for the answer."
Interview questions on this
- Why not just use whole words as tokens? The vocabulary would be huge and still miss new words, names and typos. Sub-words cover everything with a fixed vocabulary.
- Why do LLMs struggle to count the letters in a word? The model sees a few sub-word pieces, not the individual letters, so it has to have memorised how each piece is spelled.
- How would you estimate the monthly cost of a feature? Measure input and output tokens per request on a sample, multiply by prices and expected traffic, and add a margin for retries and longer conversations.
- What changes if you switch to a model with a different tokenizer? Token counts, cost, context headroom and any cached prompts. Re-measure before switching.
- What does a larger vocabulary buy you? Shorter sequences (cheaper, more fits in the window), at the price of bigger embedding and output layers.
Used in
- AI Engineering: Context Windows and Token Cost; How an LLM Picks the Next Token; Implement a BPE Tokenizer (ML coding); How Would You Cut LLM Cost by 10x?
- AI System Design: Tokens, context window and cost; Design ChatGPT; Design an LLM Gateway (token-based rate limits).