Context & memory

A model knows only what's in its context window on this call. Everything else, from the user's name to the file it read ten turns ago, is gone unless you put it back. Context engineering is deciding what goes into that window, in what order, and what gets dropped when it fills up. It matters more than prompt wording once an agent runs for more than a few turns. This lesson covers budgeting the window, compacting long conversations without breaking them, fetching context just in time, and adding long-term memory that survives between sessions.

#Why more context isn't free

Context windows are large now, often hundreds of thousands of tokens, so it's tempting to put everything in. Three costs push back.

• Money and latency. Every call re-sends the whole conversation. An agent that reads a 20,000-token file at turn 3 pays for those tokens again at turns 4, 5 and 6 through 30. Prompt caching cuts the price of the repeated prefix but not to zero, and the cost and latency lesson shows the arithmetic.
• Attention. Models get measurably worse at using information as context grows and fills with irrelevant material. A key instruction buried among 40 tool results is easier to miss than the same instruction in a short, focused context.
• Hard limits. When the total exceeds the window, the request fails. A long-running agent without a plan for this will eventually crash mid-task.

The goal isn't the smallest context. It's the smallest context that contains everything the model needs for the next step.

#Budget the window

List what goes into every call and give each part a budget, as you would for memory in a program:

Measure instead of guessing. The API can count tokens for a request without running it:

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"


def count_tokens(system: str, tools: list, messages: list) -> int:
    return client.messages.count_tokens(
        model=MODEL, system=system, tools=tools, messages=messages
    ).input_tokens

Order matters as well. Put stable content first (system prompt, tools, long reference documents) and the changing parts last (latest messages). Caching works on prefixes, so anything you edit near the start invalidates the cache for everything after it.

#Keep tool results small

Tool results are where context usually explodes. Fix it at the source.

• Return what the model needs, not what the API returned. A ticket lookup should return status, title and the last three comments, not 200 fields of JSON.
• Truncate with a note. If a file is 3,000 lines, return the first 200 and say so: [truncated: showing lines 1–200 of 3,014. Call read_file with start_line to see more.] The model can then ask for exactly the part it needs.
• Prefer search tools to dump tools. A grep(pattern) tool that returns matching lines with line numbers beats a read_all_files tool every time.

This pattern is called just-in-time context: instead of loading everything up front, give the agent lightweight handles (file paths, IDs, search tools) and let it pull in details when it needs them, the way you'd use a file browser rather than printing a whole repository.

#Compaction: summarise the past

Even with small tool results, a long session eventually fills the window. Compaction replaces the older part of the conversation with a summary and keeps recent turns verbatim.

There's a structural trap. The Messages API requires that every tool_result block follows the assistant message that contains its tool_use. If you cut the history in the middle of a tool exchange, the next request is rejected. So only cut where a plain user message (one without tool results) begins:

def is_plain_user_turn(message: dict) -> bool:
    if message["role"] != "user":
        return False
    content = message["content"]
    if isinstance(content, str):
        return True
    return not any(block.get("type") == "tool_result" for block in content)


def render(messages: list) -> str:
    """Flatten messages into plain text for the summariser."""
    lines = []
    for m in messages:
        content = m["content"]
        if isinstance(content, str):
            lines.append(f"{m['role']}: {content}")
            continue
        for b in content:
            b = b if isinstance(b, dict) else b.model_dump()
            if b["type"] == "text":
                lines.append(f"{m['role']}: {b['text']}")
            elif b["type"] == "tool_use":
                lines.append(f"tool call: {b['name']}({b['input']})")
            elif b["type"] == "tool_result":
                lines.append(f"tool result: {str(b.get('content'))[:500]}")
    return "\n".join(lines)


SUMMARY_PROMPT = """Summarise this agent session so far so the work can continue without it.
Keep: the user's goal and constraints, decisions made and why, facts discovered
(IDs, file paths, numbers), what has been done, and what is left to do.
Drop: pleasantries, dead ends that taught nothing, raw tool output."""


def compact(system: str, tools: list, messages: list, budget: int = 60_000, keep_recent: int = 8) -> list:
    if count_tokens(system, tools, messages) < budget:
        return messages
    # Find the latest safe cut point that still leaves `keep_recent` messages intact.
    cut = next(
        (i for i in range(len(messages) - keep_recent, 0, -1) if is_plain_user_turn(messages[i])),
        None,
    )
    if cut is None:
        return messages  # nothing safe to cut yet
    summary = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1500,
        system=SUMMARY_PROMPT,
        messages=[{"role": "user", "content": render(messages[:cut])}],
    ).content[0].text
    note = f"[Summary of the earlier part of this session]\n{summary}"
    first = messages[cut]
    merged = note + "\n\n" + (first["content"] if isinstance(first["content"], str) else render([first]))
    return [{"role": "user", "content": merged}, *messages[cut + 1:]]

Call compact before each model call in your agent loop. A small, fast model writes the summary, and the summary goes into the first user message so the conversation still starts with a user turn.

A cheaper first step than summarising is clearing old tool results: replace the content of tool results older than N turns with "[result cleared to save space; call the tool again if needed]". The tool_use/tool_result pairs stay intact, so the structure stays valid, and the model can re-fetch anything it needs.

#Isolate with sub-agents

Some subtasks generate a lot of context that the main task doesn't need: "read these 15 files and tell me which ones handle billing". Run that as a sub-agent with its own fresh context and its own tools, and return only its conclusion. The main agent's context grows by a paragraph instead of 15 files.

The trade-off is that the sub-agent doesn't see the main conversation, so give it a complete brief: goal, constraints, what to return and in what format. Sub-agents can also run in parallel, which helps latency when subtasks are independent.

#Long-term memory

Everything so far lives within one session. Long-term memory carries information across sessions: a user's preferences, facts about their project, what was decided last week. It's useful to think of three kinds:

• Semantic: facts. "The user's company deploys on Fly.io." "Their API is versioned under /v2."
• Episodic: past events. "On 3 March we migrated the orders table; it took two attempts."
• Procedural: how to do things. "When writing tests for this repo, use the fixtures in tests/conftest.py."

Memory has a write path and a read path. For writing, either give the agent a remember tool and let it decide what's worth saving, or run an extraction step after each session that pulls durable facts out of the transcript. For reading, load the most relevant memories into the context at the start of a session, or give the agent a recall tool.

import json
import time
from pathlib import Path

MEMORY_DIR = Path("memories")
MEMORY_TOOLS = [
    {
        "name": "remember",
        "description": (
            "Save a durable fact about the user or their project for future sessions. "
            "Use for stable preferences and facts, not for anything said only for this task. "
            "Never save secrets, passwords or tokens."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"fact": {"type": "string", "description": "One self-contained sentence."}},
            "required": ["fact"],
        },
    },
]


def load_memories(user_id: str, limit: int = 30) -> list[dict]:
    path = MEMORY_DIR / f"{user_id}.json"
    memories = json.loads(path.read_text()) if path.exists() else []
    return sorted(memories, key=lambda m: m["saved_at"], reverse=True)[:limit]


def remember(user_id: str, fact: str) -> str:
    MEMORY_DIR.mkdir(exist_ok=True)
    path = MEMORY_DIR / f"{user_id}.json"
    memories = json.loads(path.read_text()) if path.exists() else []
    memories.append({"fact": fact, "saved_at": time.time()})
    path.write_text(json.dumps(memories, indent=2))
    return "Saved."


def system_prompt(user_id: str) -> str:
    facts = "\n".join(f"- {m['fact']}" for m in load_memories(user_id))
    return (
        "You are a coding assistant.\n\n"
        "What you remember about this user from earlier sessions (may be out of date; "
        "if it conflicts with what they say now, trust what they say now):\n" + (facts or "- nothing yet")
    )

Loading the newest 30 facts is fine for a handful of memories per user. Once there are hundreds, switch the read path to the retrieval you built in the RAG lesson: embed each memory and fetch the ones closest to the current task.

#What goes wrong with memory

• Stale facts. "Uses Python 3.10" was true in January. Store timestamps, let newer facts replace older ones, and tell the model memories may be outdated.
• Junk accumulation. Agents over-remember. Review what's being saved, deduplicate, and cap the number per user.
• Privacy. Memory is personal data. Scope it strictly per user (never let one user's memories reach another's context), never store secrets, and give users a way to see and delete what's remembered.
• Persistent injection. If an attacker gets text into a document the agent reads, and the agent "remembers" an instruction from it, the attack now applies to every future session. Store facts the user stated, not instructions found in content, and treat memory as data rather than commands.

#Practice

Take the agent loop from the agents lesson (or the MCP client from the previous lesson) and add context management:

1. Before each call, print the token count from count_tokens.
2. Make every tool result longer than 2,000 characters truncate with a note telling the model how to get more.
3. Add compact with a deliberately low budget (for example 4,000 tokens) and verify, across a 10-turn session, that requests never fail and that the agent still remembers the original goal.
4. Add the remember tool and the memory-aware system prompt, then start a second session and check that a preference you stated in the first is used.

MAX_RESULT_CHARS = 2000


def truncate(text: str) -> str:
    if len(text) <= MAX_RESULT_CHARS:
        return text
    return (
        text[:MAX_RESULT_CHARS]
        + f"\n[truncated: showed {MAX_RESULT_CHARS} of {len(text)} characters. "
        "Ask for a narrower range or a search instead of the whole thing.]"
    )


def run(user_id: str, task: str, tools: list, execute) -> str:
    system = system_prompt(user_id)
    all_tools = tools + MEMORY_TOOLS
    messages: list = [{"role": "user", "content": task}]
    for _ in range(25):
        messages = compact(system, all_tools, messages, budget=4_000, keep_recent=6)
        print(f"[context: {count_tokens(system, all_tools, messages)} tokens]")
        response = client.messages.create(
            model=MODEL, max_tokens=2048, system=system, tools=all_tools, messages=messages
        )
        messages.append({"role": "assistant", "content": response.content})
        if response.stop_reason != "tool_use":
            return "".join(b.text for b in response.content if b.type == "text")
        results = []
        for block in response.content:
            if block.type != "tool_use":
                continue
            if block.name == "remember":
                output = remember(user_id, block.input["fact"])
            else:
                output = truncate(execute(block.name, block.input))
            results.append({"type": "tool_result", "tool_use_id": block.id, "content": output})
        messages.append({"role": "user", "content": results})
    return "Stopped: step limit reached."

#Key takeaways

• The context window is the model's only memory for a call; decide deliberately what goes in, in what order, and what leaves.
• Measure with token counting, keep stable content first for caching, and shrink tool results at the source.
• Compact by summarising older turns, but only cut where a plain user turn begins so tool_use and tool_result stay paired.
• Use just-in-time retrieval and sub-agents to keep bulky material out of the main context.
• Long-term memory needs a write path, a read path, timestamps, per-user isolation and protection against remembered injections.