Prompt caching

Before generating anything, a model processes every input token and builds internal state for it, the KV cache. If the next request starts with the same prefix (the same system prompt, tool definitions and documents), that work can be reused instead of redone. Prompt caching exposes this reuse: cached input tokens are billed at a fraction of the normal price, and time-to-first-token drops, often a lot for long prompts.

#How it works

• Prefix match only. The cache covers the request from the first token up to a cache point. Change one token early in the prompt and everything after it is a miss.
• Explicit or automatic. Some APIs cache automatically above a minimum length. Others let you mark breakpoints, for example a cache_control marker on a content block, and charge a small premium to write the cache.
• Short-lived. Entries expire after minutes of inactivity unless they're used again, with longer lifetimes available on some APIs.
• Exact and per-organization. Caches aren't shared across accounts, and hits are reported in the response's usage fields.

#Structure prompts for it

Order content from most stable to most variable:

[ tools ] [ system prompt ] [ reference docs / examples ]  ◂ cache point
[ conversation history ]                                   ◂ cache point (moves each turn)
[ the new user message ]

• Keep the prefix byte-for-byte stable. Don't put a timestamp, request ID or the user's name in the system prompt. Put variable data after the cached part.
• Keep tool definitions in a fixed order. Reordering them changes the prefix.
• In multi-turn chats and agents, move the cache point forward each turn so the growing history is reused too. That's where the biggest savings are, because every turn resends everything before it. See the tool-use loop.

#When it pays off

• Long system prompts or tool sets reused across many requests.
• Chatting with a large document: cache the document and vary the question.
• Agents that run many steps over the same context.
• Many few-shot examples.

It does little for short, one-off prompts, or when every request has a unique prefix. Check the cache-read and cache-write token counts in production. A hit rate near zero almost always means something variable crept into the prefix.

This is different from caching responses in your application, which skips the model call entirely but only helps when the whole request repeats. See caching patterns for that.