Context windows

A model has no memory between API calls. Everything it can use on a call must be in that call's context window: the system prompt, tool definitions, conversation history, any documents you paste in, and the output it generates. All of it is measured in tokens, and the window has a hard maximum, from tens of thousands to over a million tokens depending on the model.

┌──────────────── context window ────────────────┐
│ system prompt │ tools │ history │ documents │ ▸ output (max_tokens) │
└───────────────────────────────────────────────────┘

If the input plus the max_tokens you reserve for the answer exceeds the window, the request fails, or the answer gets cut off when output hits its limit.

#Bigger isn't free

• Cost and latency grow with input. You pay for every input token on every call. A chat that resends a 50,000-token history each turn is expensive unless you use prompt caching.
• Attention gets diluted. Models are better at using information near the start and end of a long context than information buried in the middle ("lost in the middle"), and quality on details drops as context grows. A focused prompt usually beats a stuffed one.
• Long conversations accumulate junk: old tool results, superseded plans, wrong guesses. They all compete for attention.

#Managing it

• Retrieve instead of stuffing. Put the few relevant passages in the context, not the whole corpus. That's what RAG and chunking are for.
• Put stable content first. Instructions and reference documents go at the top, which also makes them cacheable. Put the question at the end, close to where the model starts writing.
• Trim or summarize history. Keep the last few turns verbatim, replace older ones with a summary, and drop bulky tool output once it's been used.
• Reserve output space deliberately. Set max_tokens to what the task needs, and check the stop reason so you can tell a finished answer from one that hit the limit.
• For agents, keep state outside the window. Write notes, files or task lists the model can re-read, instead of relying on everything staying in context.

Treat the window like RAM: large but finite, with every byte paid for on every call.