# Tokens and tokenization

> Models read and bill in sub-word tokens, not characters or words — and that explains a lot of odd behaviour.

Concept · Models & prompting · https://levelcode.org/concepts/tokens

A language model never sees characters. A **tokenizer** first splits the text into **tokens**, which are chunks from a fixed vocabulary of about 50,000–250,000 entries, and turns each one into an integer ID. The model reads and predicts those IDs.

```text
"Tokenization isn't magic."
→ ["Token", "ization", " isn", "'t", " magic", "."]   (illustrative; every model's tokenizer differs)
```

Most tokenizers use **byte-pair encoding (BPE)** or a close relative. They start from bytes and repeatedly merge the most frequent adjacent pairs in a large training corpus. Common words become a single token. Rare words, names, code identifiers and non-English text get split into several pieces.

## Rules of thumb

- **English prose:** roughly 3–4 characters, or about ¾ of a word, per token. 1,000 tokens is about 750 words.
- **Code, JSON, URLs and numbers** use more tokens per character than prose.
- **Many non-Latin scripts** use several times more tokens than English for the same meaning, which means they cost more and leave less room.
- Leading spaces are usually part of the token, so `" magic"` and `"magic"` are different tokens.

Count tokens with the provider's token-counting endpoint or the model's own tokenizer. Different model families tokenize the same text differently.

## Why it matters

- **Cost and limits are in tokens.** You pay per input and output token, and the [context window](https://levelcode.org/concepts/context-windows) is a token budget. Output tokens usually cost several times more than input tokens.
- **Latency follows output length.** The model generates one token at a time, so a 1,000-token answer takes about ten times as long as a 100-token one. Asking for shorter output is often the biggest speed-up available.
- **It explains odd mistakes.** Counting the letters in "strawberry," reversing a string, or exact arithmetic on long numbers is hard because the model sees a few chunks, not individual characters or digits.
- **Repeated prefixes are reusable.** Identical leading tokens across requests can be served from a [prompt cache](https://levelcode.org/concepts/prompt-caching) for a fraction of the price.

Each new token is chosen by [sampling](https://levelcode.org/concepts/sampling-temperature) from the model's probabilities over this same vocabulary. The [how LLMs work lesson](https://levelcode.org/roadmaps/ai-engineer/how-llms-work) puts the pieces together.