Concept · Models & prompting
Tokens and tokenization
Models read and bill in sub-word tokens, not characters or words — and that explains a lot of odd behaviour.
A language model never sees characters. A tokenizer first splits the text into tokens, which are chunks from a fixed vocabulary of about 50,000–250,000 entries, and turns each one into an integer ID. The model reads and predicts those IDs.
"Tokenization isn't magic."
→ ["Token", "ization", " isn", "'t", " magic", "."] (illustrative; every model's tokenizer differs)Most tokenizers use byte-pair encoding (BPE) or a close relative. They start from bytes and repeatedly merge the most frequent adjacent pairs in a large training corpus. Common words become a single token. Rare words, names, code identifiers and non-English text get split into several pieces.
Rules of thumb
- English prose: roughly 3–4 characters, or about ¾ of a word, per token. 1,000 tokens is about 750 words.
- Code, JSON, URLs and numbers use more tokens per character than prose.
- Many non-Latin scripts use several times more tokens than English for the same meaning, which means they cost more and leave less room.
- Leading spaces are usually part of the token, so
" magic"and"magic"are different tokens.
Count tokens with the provider's token-counting endpoint or the model's own tokenizer. Different model families tokenize the same text differently.
Why it matters
- Cost and limits are in tokens. You pay per input and output token, and the context window is a token budget. Output tokens usually cost several times more than input tokens.
- Latency follows output length. The model generates one token at a time, so a 1,000-token answer takes about ten times as long as a 100-token one. Asking for shorter output is often the biggest speed-up available.
- It explains odd mistakes. Counting the letters in "strawberry," reversing a string, or exact arithmetic on long numbers is hard because the model sees a few chunks, not individual characters or digits.
- Repeated prefixes are reusable. Identical leading tokens across requests can be served from a prompt cache for a fraction of the price.
Each new token is chosen by sampling from the model's probabilities over this same vocabulary. The how LLMs work lesson puts the pieces together.