# How LLMs work > Explain tokens, next-token prediction, training, context windows, hallucination and thinking, and pick a model on capability, latency and cost. AI Engineer roadmap · LLM foundations · 18 min read · https://levelcode.org/roadmaps/ai-engineer/how-llms-work You will build better AI features if you have an accurate picture of what a large language model (LLM) actually does. It isn't a database, a search engine or a mind. It is a function that reads a sequence of tokens and predicts the next one, over and over. This lesson covers tokenization, next-token prediction, how models are trained and run, context windows, why models make things up, what "thinking" adds, and how to choose a model. ## Tokens: what the model actually reads A model never sees characters or words. Before your text reaches it, a **tokenizer** splits the text into **tokens** and maps each one to an integer ID from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Most modern tokenizers use a subword scheme (usually a variant of byte-pair encoding): common words become a single token, and rare words are split into pieces. An illustrative split (real boundaries depend on the tokenizer) looks like this: ```text "The cat sat." -> ["The", " cat", " sat", "."] 4 tokens "unbelievably" -> ["un", "believ", "ably"] 3 tokens "Kubernetes 1.31" -> ["Kub", "ernetes", " ", "1", ".", "31"] 6 tokens ``` Leading spaces are usually part of the token, and code, numbers and non-English text use more tokens per character than English prose. A rough rule for English is about four characters, or three quarters of a word, per token. Tokenizers differ between model families, so measure when it matters: Anthropic's API has a free token-counting endpoint, which you will use in the next lesson. Try it below. Type your own text or pick a preset, and compare the characters-per-token figure for English prose with code, a long number, a German compound word and emoji. Watch where the boundaries fall and how often a leading space (shown as `·`) belongs to the token after it. > **Interactive: Text to tokens, with a toy vocabulary.** Common English words become one token each, while digits, punctuation-heavy code and characters the vocabulary lacks break into many small pieces. Real tokenizers split differently, but you pay for the same pattern. Try it: https://levelcode.org/labs/tokenizer Tokens matter for three practical reasons: 1. **You pay per token**, for input and output separately. 2. **Limits are in tokens.** Context windows and output caps are token counts, not word counts. 3. **Character-level tasks are harder than they look.** Asked how many letters are in a word, the model has to reason about characters it never directly saw. That is why models historically fumbled "count the r's in strawberry" while writing fluent essays. ## Next-token prediction At its core, an LLM is a function: given the tokens so far, it outputs a probability for every token in the vocabulary being next. To generate text, the system picks one token from that distribution, appends it, and runs the model again. This loop is called **autoregressive generation**. You can see the idea with a toy model that just counts which word follows which in a tiny corpus: ```python import random from collections import Counter, defaultdict corpus = "the cat sat on the mat . the cat ate the fish . the dog sat on the rug ." tokens = corpus.split() counts: dict[str, Counter[str]] = defaultdict(Counter) for prev, nxt in zip(tokens, tokens[1:]): counts[prev][nxt] += 1 def next_token_probs(prev: str) -> dict[str, float]: c = counts[prev] total = sum(c.values()) return {tok: round(n / total, 3) for tok, n in c.most_common()} def generate(start: str, n: int, greedy: bool = False) -> str: out = [start] for _ in range(n): probs = next_token_probs(out[-1]) if not probs: break if greedy: out.append(max(probs, key=probs.get)) else: out.append(random.choices(list(probs), weights=list(probs.values()))[0]) return " ".join(out) print(next_token_probs("the")) print(generate("the", 8, greedy=True)) print(generate("the", 8)) # sampled: differs between runs ``` ```text {'cat': 0.333, 'mat': 0.167, 'fish': 0.167, 'dog': 0.167, 'rug': 0.167} the cat sat on the cat sat on the the cat ate the cat sat on the dog ``` Two lessons carry over to real models. First, **always picking the most likely token (greedy decoding) tends to loop and sound flat**, so production systems *sample* from the distribution instead. Second, **sampling means the same prompt can produce different outputs.** Some APIs and open-model runtimes expose knobs such as `temperature` that sharpen or flatten the distribution. Current Claude models manage sampling themselves and reject such parameters, so you shape behaviour through instructions, examples and structured output instead. The counting toy skips a step that real models have. A model doesn't output probabilities directly: it outputs a score called a **logit** for every token in its vocabulary, and the **softmax** function turns those scores into probabilities that sum to 1. Temperature `T` divides the logits before softmax, so `T` below 1 sharpens the distribution and `T` above 1 flattens it. In the widget below, drag the temperature slider and watch the bars, then sample a few tokens. With a fixed seed the same draws repeat; switch to greedy and you always get the starred token, whatever the temperature. > **Interactive: Predicting the next token.** Try it: raise the temperature on the France prompt and watch ' Lyon', a fluent wrong answer, get a real chance of being drawn. Try it: https://levelcode.org/labs/next-token A real LLM differs from the toy in one crucial way. Instead of a lookup table keyed on the previous word, it is a neural network (a **transformer**) with billions of learned parameters that conditions on *every* token in its context at once. Its attention mechanism lets each position draw on any earlier one, which is how it resolves "it" to a noun three paragraphs back. Because it learns general patterns rather than memorising counts, it can also continue text it has never seen. ## Training versus inference **Pre-training** produces the raw capability. The model is trained to predict the next token across an enormous corpus of text and code, and to do that well it has to absorb grammar, facts, coding conventions and patterns of reasoning. The result is a *base model*: a superb text continuer, but not yet a helpful assistant. **Post-training** turns it into one. The model is fine-tuned on examples of instructions and good responses, then further trained with reinforcement learning on feedback: human preference judgements, AI feedback against written principles, and automatically checkable tasks like passing unit tests. This is where instruction following, refusals of harmful requests, tool use and the assistant's style come from. **Inference** is what happens when you call an API. The weights are frozen, and each request is processed independently, so the model has **no memory between calls**. A chat product that "remembers" you re-sends earlier messages, or a summary of them, in every request. Every model also has a **knowledge cutoff**, the date its training data ends. It doesn't know about later events, and it may not know that it doesn't know. Inference has two phases with different costs: | Phase | What happens | Consequence | | --- | --- | --- | | Prefill | All input tokens are processed in parallel | Fast and cheaper per token; long prompts add some latency | | Decode | Output tokens are generated one at a time, each needing a full model pass | Slower and pricier per token; latency grows with output length | That asymmetry is why output tokens cost several times more than input tokens, and why "respond in under 100 words" is one of the most effective latency optimisations you have. ## The context window The **context window** is the maximum number of tokens the model can consider in one request: system prompt, conversation history, documents, tool definitions *and* the output it generates. Current Claude models have a 1M-token context window and can produce up to 128K output tokens in a response. Other models range from a few thousand tokens to millions. Everything the model knows about your situation must be inside that window. If a fact isn't in the context and wasn't learned in training, the model doesn't have it. That gives you the central design principle of AI engineering: **put the information the model needs into the context**, and leave out what it doesn't need. A large window is not a reason to fill it. Cost and latency grow with input size, and models can miss or under-weight a detail buried in a very long context. A focused 5K-token prompt usually beats a 500K-token dump. > [!NOTE] > When a conversation outgrows its window, something has to give. Applications summarise older turns, retrieve only the relevant documents (the RAG lesson covers this), or start fresh. The API won't silently "remember" what you dropped. ## Why models hallucinate A **hallucination** is fluent, confident output that is false: an invented API method, a fake citation, a wrong date. It follows from how models work. They are trained to produce *plausible continuations*, and plausible and true usually coincide, but not always. There is no separate fact store to check against. Hallucinations are most likely when: - the question concerns **rare or very specific facts**: exact figures, minor people, version-specific APIs, URLs and paper titles; - the answer post-dates the **knowledge cutoff**; - the prompt **presupposes something false** ("Why did Python 4 remove the GIL?"); - the model is asked to produce something it **cannot know**, like the contents of a document it was never shown. Modern models are much better at saying "I'm not sure", but engineer around the risk rather than hoping it away: 1. **Ground the model.** Supply the source documents and ask it to answer only from them. This is the core idea of retrieval-augmented generation. 2. **Give it permission to not know.** "If the answer isn't in the provided documents, say so" measurably reduces fabrication. 3. **Give it tools** to look things up or compute rather than recall: search, a database query, a calculator. 4. **Verify what matters.** Check generated code by running tests, check citations against sources, and validate extracted fields. ## What thinking adds Many current models can **think** before answering: they generate intermediate reasoning tokens, then the final response. Thinking uses the same next-token process, so it isn't a different kind of intelligence. It gives the model more computation and a scratchpad. Working through sub-steps and checking its own work improves accuracy on multi-step problems such as maths, debugging, planning and careful analysis. Thinking has a price. Thinking tokens are output tokens: you pay for them, and they add latency before the first visible word. It also doesn't guarantee correctness, because a model can reason fluently to a wrong conclusion. Current Claude models use **adaptive thinking**: the model decides how much to think for each request, and you steer the depth with an **effort** setting from `low` to `max`. Low effort suits simple classification or extraction; high effort suits hard problems where accuracy beats speed. By default, Claude doesn't return the raw reasoning text. You can opt in to a summary of it. ## Choosing a model Model choice is a three-way trade between **capability**, **latency** and **cost**. Anthropic's current family illustrates the tiers that most providers offer: | Model | Input / output price per 1M tokens | Good for | | --- | --- | --- | | `claude-opus-5-5` | $4 / $20 | Hardest reasoning, long agentic tasks, complex code | | `claude-sonnet-5-5` | $2 / $10 | Strong all-rounder for high-volume production work | | `claude-haiku-5-5` | $0.10 / $0.50 (prompts up to 100K tokens) | Classification, routing, extraction, latency-sensitive tasks | Here is a worked cost estimate. A support assistant handles 10,000 requests a day, each with 3,000 input tokens and 400 output tokens: ```text Input: 10,000 x 3,000 = 30,000,000 tokens = 30 M Output: 10,000 x 400 = 4,000,000 tokens = 4 M Opus 5.5: 30 x $4.00 + 4 x $20.00 = $120 + $80 = $200 / day Sonnet 5.5: 30 x $2.00 + 4 x $10.00 = $60 + $40 = $100 / day Haiku 5.5: 30 x $0.10 + 4 x $0.50 = $3 + $2 = $5 / day ``` Thinking tokens would add to the output side. With a 40x price gap between tiers, a practical process is: 1. **Prototype with a capable model** to learn what's possible. 2. **Build a small evaluation set** of real inputs with known-good outputs (the evals lesson covers this). 3. **Try cheaper and faster models** against it. Use the cheapest one that clears your quality bar, and route only the hard cases to the expensive one. > [!TIP] > Models and prices change every few months. Keep the model ID in configuration rather than scattered through your code, so trying a newer or cheaper model is a one-line change you can test against your evaluation set. ### Open versus closed models **Closed models** such as Claude are available only through an API: frontier capability with no infrastructure to run, but you depend on the provider's availability, pricing and data policies. **Open-weight models** (families such as Llama, Mistral, Qwen and Gemma) publish their weights, so you can run and fine-tune them on your own hardware and keep data in-house. In exchange, you own GPUs, scaling and updates, and the strongest open models generally trail the frontier on the hardest tasks. "Open weights" isn't open source: training data is rarely released, and licences vary. Many teams start on a hosted API and move specific workloads to open models later, as the self-hosting lesson describes. ## Practice Copy the toy bigram model above into `toy.py` and run it. Then: 1. Change it into a **trigram** model, where the next token depends on the previous *two* tokens. Generate a few sentences starting from `("the", "dog")`. 2. Print `next_token_probs(("the", "bird"))`. Explain in one sentence why a neural LLM behaves differently on unseen context. 3. Estimate the monthly cost (30 days) of a summarisation feature on `claude-sonnet-5-5`: 2,000 requests a day, 6,000 input tokens and 300 output tokens each. **Solution** ```python import random from collections import Counter, defaultdict corpus = "the cat sat on the mat . the cat ate the fish . the dog sat on the rug ." tokens = corpus.split() counts: dict[tuple[str, str], Counter[str]] = defaultdict(Counter) for a, b, nxt in zip(tokens, tokens[1:], tokens[2:]): counts[(a, b)][nxt] += 1 def next_token_probs(context: tuple[str, str]) -> dict[str, float]: c = counts[context] total = sum(c.values()) return {tok: n / total for tok, n in c.most_common()} def generate(a: str, b: str, n: int) -> str: out = [a, b] for _ in range(n): probs = next_token_probs((out[-2], out[-1])) if not probs: break out.append(random.choices(list(probs), weights=list(probs.values()))[0]) return " ".join(out) print(next_token_probs(("on", "the"))) print(next_token_probs(("the", "bird"))) for _ in range(3): print(generate("the", "dog", 8)) ``` ```text {'mat': 0.5, 'rug': 0.5} {} the dog sat on the rug . the dog sat on the mat . the cat ate the dog sat on the rug . ``` Your samples will vary. More context makes output more coherent but closer to copying the corpus. Generation stops at `rug .` because nothing ever followed that pair. **Unseen context:** the counting model returns nothing for `("the", "bird")` because it can only replay exact sequences it has seen. A neural LLM maps tokens into learned representations where "bird" sits near other animals, so it generalises to contexts it never saw verbatim. **Cost:** ```text Input: 2,000 x 6,000 x 30 = 360 M tokens x $2.00 = $720 Output: 2,000 x 300 x 30 = 18 M tokens x $10.00 = $180 Total ≈ $900 / month (before any thinking tokens or caching discounts) ``` Input dominates here because summarisation reads a lot and writes a little, so trimming the prompt or prompt caching saves the most. ## Key takeaways - Models read **tokens**, not characters. Cost, limits and some odd failures follow from that. - An LLM predicts a probability distribution over the next token and **samples** from it repeatedly, so outputs vary between runs. - Pre-training builds capability, post-training makes an assistant, and at inference the weights are frozen and **every request is stateless**. - Everything the model needs must fit in the **context window**, and more context isn't free or automatically better. - **Hallucination** is a natural failure of plausible-continuation generation; counter it with grounding, permission to say "I don't know", tools and verification. - Choose models empirically: prototype with a strong model, evaluate, then use the cheapest one that meets the bar. ## Quiz 1. Why are output tokens typically priced several times higher than input tokens? A. Output tokens are longer than input tokens B. Input tokens are processed in parallel during prefill, while output tokens are generated one at a time, each needing its own pass through the model C. Providers store output tokens permanently D. Output tokens go through a separate, larger model **Answer: B.** Prefill handles the whole prompt in parallel. Decoding produces one token per forward pass, which uses far more compute and time per token. The same asymmetry is why long outputs add latency. 2. A user tells your chatbot their name, and in the next request the model knows it. What is actually happening? A. The model updated its weights during the first request B. The provider keeps a hidden per-user memory inside the model C. Your application re-sent the earlier messages as part of the new request's context D. The model inferred the name from the user's IP address **Answer: C.** Inference is stateless and the weights are frozen. Any memory across turns exists because the application includes previous messages, or a summary of them, in each new request. 3. Which change most directly reduces hallucinated answers about your company's refund policy? A. Including the policy text in the prompt and instructing the model to answer only from it, saying so if the answer isn't there B. Switching to a model with a larger context window C. Asking the model to be more confident D. Adding 'Do not hallucinate' in capital letters **Answer: A.** Grounding the model in the source document, and giving it explicit permission to say it doesn't know, addresses the cause: without the policy in context, the model can only produce a plausible guess. A bigger window does nothing if the policy still isn't in it. 4. What does enabling deeper thinking (higher effort) trade off? A. Nothing; it is strictly better B. It improves speed but lowers accuracy C. It shrinks the context window D. It usually improves accuracy on multi-step problems, at the cost of more output tokens and higher latency **Answer: D.** Thinking gives the model extra computation as intermediate tokens. You pay for those tokens and wait for them, so reserve deeper thinking for tasks that need it, and keep low effort for simple classification or extraction.