How LLMs work

You will build better AI features if you have an accurate picture of what a large language model (LLM) actually does. It isn't a database, a search engine or a mind. It is a function that reads a sequence of tokens and predicts the next one, over and over. This lesson covers tokenization, next-token prediction, how models are trained and run, context windows, why models make things up, what "thinking" adds, and how to choose a model.

#Tokens: what the model actually reads

A model never sees characters or words. Before your text reaches it, a tokenizer splits the text into tokens and maps each one to an integer ID from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Most modern tokenizers use a subword scheme (usually a variant of byte-pair encoding): common words become a single token, and rare words are split into pieces.

An illustrative split (real boundaries depend on the tokenizer) looks like this:

"The cat sat."      ->  ["The", " cat", " sat", "."]                  4 tokens
"unbelievably"      ->  ["un", "believ", "ably"]                      3 tokens
"Kubernetes 1.31"   ->  ["Kub", "ernetes", " ", "1", ".", "31"]        6 tokens

Leading spaces are usually part of the token, and code, numbers and non-English text use more tokens per character than English prose. A rough rule for English is about four characters, or three quarters of a word, per token. Tokenizers differ between model families, so measure when it matters: Anthropic's API has a free token-counting endpoint, which you will use in the next lesson.

Try it below. Type your own text or pick a preset, and compare the characters-per-token figure for English prose with code, a long number, a German compound word and emoji. Watch where the boundaries fall and how often a leading space (shown as ·) belongs to the token after it.

Tokens matter for three practical reasons:

1. You pay per token, for input and output separately.
2. Limits are in tokens. Context windows and output caps are token counts, not word counts.
3. Character-level tasks are harder than they look. Asked how many letters are in a word, the model has to reason about characters it never directly saw. That is why models historically fumbled "count the r's in strawberry" while writing fluent essays.

#Next-token prediction

At its core, an LLM is a function: given the tokens so far, it outputs a probability for every token in the vocabulary being next. To generate text, the system picks one token from that distribution, appends it, and runs the model again. This loop is called autoregressive generation.

You can see the idea with a toy model that just counts which word follows which in a tiny corpus:

import random
from collections import Counter, defaultdict

corpus = "the cat sat on the mat . the cat ate the fish . the dog sat on the rug ."
tokens = corpus.split()

counts: dict[str, Counter[str]] = defaultdict(Counter)
for prev, nxt in zip(tokens, tokens[1:]):
    counts[prev][nxt] += 1

def next_token_probs(prev: str) -> dict[str, float]:
    c = counts[prev]
    total = sum(c.values())
    return {tok: round(n / total, 3) for tok, n in c.most_common()}

def generate(start: str, n: int, greedy: bool = False) -> str:
    out = [start]
    for _ in range(n):
        probs = next_token_probs(out[-1])
        if not probs:
            break
        if greedy:
            out.append(max(probs, key=probs.get))
        else:
            out.append(random.choices(list(probs), weights=list(probs.values()))[0])
    return " ".join(out)

print(next_token_probs("the"))
print(generate("the", 8, greedy=True))
print(generate("the", 8))  # sampled: differs between runs

{'cat': 0.333, 'mat': 0.167, 'fish': 0.167, 'dog': 0.167, 'rug': 0.167}
the cat sat on the cat sat on the
the cat ate the cat sat on the dog

Two lessons carry over to real models. First, always picking the most likely token (greedy decoding) tends to loop and sound flat, so production systems sample from the distribution instead. Second, sampling means the same prompt can produce different outputs. Some APIs and open-model runtimes expose knobs such as temperature that sharpen or flatten the distribution. Current Claude models manage sampling themselves and reject such parameters, so you shape behaviour through instructions, examples and structured output instead.

The counting toy skips a step that real models have. A model doesn't output probabilities directly: it outputs a score called a logit for every token in its vocabulary, and the softmax function turns those scores into probabilities that sum to 1. Temperature T divides the logits before softmax, so T below 1 sharpens the distribution and T above 1 flattens it. In the widget below, drag the temperature slider and watch the bars, then sample a few tokens. With a fixed seed the same draws repeat; switch to greedy and you always get the starred token, whatever the temperature.

A real LLM differs from the toy in one crucial way. Instead of a lookup table keyed on the previous word, it is a neural network (a transformer) with billions of learned parameters that conditions on every token in its context at once. Its attention mechanism lets each position draw on any earlier one, which is how it resolves "it" to a noun three paragraphs back. Because it learns general patterns rather than memorising counts, it can also continue text it has never seen.

#Training versus inference

Pre-training produces the raw capability. The model is trained to predict the next token across an enormous corpus of text and code, and to do that well it has to absorb grammar, facts, coding conventions and patterns of reasoning. The result is a base model: a superb text continuer, but not yet a helpful assistant.

Post-training turns it into one. The model is fine-tuned on examples of instructions and good responses, then further trained with reinforcement learning on feedback: human preference judgements, AI feedback against written principles, and automatically checkable tasks like passing unit tests. This is where instruction following, refusals of harmful requests, tool use and the assistant's style come from.

Inference is what happens when you call an API. The weights are frozen, and each request is processed independently, so the model has no memory between calls. A chat product that "remembers" you re-sends earlier messages, or a summary of them, in every request. Every model also has a knowledge cutoff, the date its training data ends. It doesn't know about later events, and it may not know that it doesn't know.

Inference has two phases with different costs:

That asymmetry is why output tokens cost several times more than input tokens, and why "respond in under 100 words" is one of the most effective latency optimisations you have.

#The context window

The context window is the maximum number of tokens the model can consider in one request: system prompt, conversation history, documents, tool definitions and the output it generates. Current Claude models have a 1M-token context window and can produce up to 128K output tokens in a response. Other models range from a few thousand tokens to millions.

Everything the model knows about your situation must be inside that window. If a fact isn't in the context and wasn't learned in training, the model doesn't have it. That gives you the central design principle of AI engineering: put the information the model needs into the context, and leave out what it doesn't need.

A large window is not a reason to fill it. Cost and latency grow with input size, and models can miss or under-weight a detail buried in a very long context. A focused 5K-token prompt usually beats a 500K-token dump.

#Why models hallucinate

A hallucination is fluent, confident output that is false: an invented API method, a fake citation, a wrong date. It follows from how models work. They are trained to produce plausible continuations, and plausible and true usually coincide, but not always. There is no separate fact store to check against.

Hallucinations are most likely when:

• the question concerns rare or very specific facts: exact figures, minor people, version-specific APIs, URLs and paper titles;
• the answer post-dates the knowledge cutoff;
• the prompt presupposes something false ("Why did Python 4 remove the GIL?");
• the model is asked to produce something it cannot know, like the contents of a document it was never shown.

Modern models are much better at saying "I'm not sure", but engineer around the risk rather than hoping it away:

1. Ground the model. Supply the source documents and ask it to answer only from them. This is the core idea of retrieval-augmented generation.
2. Give it permission to not know. "If the answer isn't in the provided documents, say so" measurably reduces fabrication.
3. Give it tools to look things up or compute rather than recall: search, a database query, a calculator.
4. Verify what matters. Check generated code by running tests, check citations against sources, and validate extracted fields.

#What thinking adds

Many current models can think before answering: they generate intermediate reasoning tokens, then the final response. Thinking uses the same next-token process, so it isn't a different kind of intelligence. It gives the model more computation and a scratchpad. Working through sub-steps and checking its own work improves accuracy on multi-step problems such as maths, debugging, planning and careful analysis.

Thinking has a price. Thinking tokens are output tokens: you pay for them, and they add latency before the first visible word. It also doesn't guarantee correctness, because a model can reason fluently to a wrong conclusion. Current Claude models use adaptive thinking: the model decides how much to think for each request, and you steer the depth with an effort setting from low to max. Low effort suits simple classification or extraction; high effort suits hard problems where accuracy beats speed. By default, Claude doesn't return the raw reasoning text. You can opt in to a summary of it.

#Choosing a model

Model choice is a three-way trade between capability, latency and cost. Anthropic's current family illustrates the tiers that most providers offer:

Here is a worked cost estimate. A support assistant handles 10,000 requests a day, each with 3,000 input tokens and 400 output tokens:

Input:  10,000 x 3,000 = 30,000,000 tokens = 30 M
Output: 10,000 x   400 =  4,000,000 tokens =  4 M

Opus 5.5:   30 x $4.00 + 4 x $20.00 = $120 + $80 = $200 / day
Sonnet 5.5: 30 x $2.00 + 4 x $10.00 = $60  + $40 = $100 / day
Haiku 5.5:  30 x $0.10 + 4 x $0.50  = $3   + $2  = $5   / day

Thinking tokens would add to the output side. With a 40x price gap between tiers, a practical process is:

1. Prototype with a capable model to learn what's possible.
2. Build a small evaluation set of real inputs with known-good outputs (the evals lesson covers this).
3. Try cheaper and faster models against it. Use the cheapest one that clears your quality bar, and route only the hard cases to the expensive one.

#Open versus closed models

Closed models such as Claude are available only through an API: frontier capability with no infrastructure to run, but you depend on the provider's availability, pricing and data policies. Open-weight models (families such as Llama, Mistral, Qwen and Gemma) publish their weights, so you can run and fine-tune them on your own hardware and keep data in-house. In exchange, you own GPUs, scaling and updates, and the strongest open models generally trail the frontier on the hardest tasks. "Open weights" isn't open source: training data is rarely released, and licences vary. Many teams start on a hosted API and move specific workloads to open models later, as the self-hosting lesson describes.

#Practice

Copy the toy bigram model above into toy.py and run it. Then:

1. Change it into a trigram model, where the next token depends on the previous two tokens. Generate a few sentences starting from ("the", "dog").
2. Print next_token_probs(("the", "bird")). Explain in one sentence why a neural LLM behaves differently on unseen context.
3. Estimate the monthly cost (30 days) of a summarisation feature on claude-sonnet-5-5: 2,000 requests a day, 6,000 input tokens and 300 output tokens each.

import random
from collections import Counter, defaultdict

corpus = "the cat sat on the mat . the cat ate the fish . the dog sat on the rug ."
tokens = corpus.split()

counts: dict[tuple[str, str], Counter[str]] = defaultdict(Counter)
for a, b, nxt in zip(tokens, tokens[1:], tokens[2:]):
    counts[(a, b)][nxt] += 1

def next_token_probs(context: tuple[str, str]) -> dict[str, float]:
    c = counts[context]
    total = sum(c.values())
    return {tok: n / total for tok, n in c.most_common()}

def generate(a: str, b: str, n: int) -> str:
    out = [a, b]
    for _ in range(n):
        probs = next_token_probs((out[-2], out[-1]))
        if not probs:
            break
        out.append(random.choices(list(probs), weights=list(probs.values()))[0])
    return " ".join(out)

print(next_token_probs(("on", "the")))
print(next_token_probs(("the", "bird")))
for _ in range(3):
    print(generate("the", "dog", 8))

{'mat': 0.5, 'rug': 0.5}
{}
the dog sat on the rug .
the dog sat on the mat . the cat ate
the dog sat on the rug .

Input:  2,000 x 6,000 x 30 = 360 M tokens x $2.00  = $720
Output: 2,000 x   300 x 30 =  18 M tokens x $10.00 = $180
Total ≈ $900 / month (before any thinking tokens or caching discounts)

#Key takeaways

• Models read tokens, not characters. Cost, limits and some odd failures follow from that.
• An LLM predicts a probability distribution over the next token and samples from it repeatedly, so outputs vary between runs.
• Pre-training builds capability, post-training makes an assistant, and at inference the weights are frozen and every request is stateless.
• Everything the model needs must fit in the context window, and more context isn't free or automatically better.
• Hallucination is a natural failure of plausible-continuation generation; counter it with grounding, permission to say "I don't know", tools and verification.
• Choose models empirically: prototype with a strong model, evaluate, then use the cheapest one that meets the bar.