# Prompt caching

> Reuse the model’s work on an unchanged prompt prefix for a fraction of the price and latency.

Concept · LLMs in production · https://levelcode.org/concepts/prompt-caching

Before generating anything, a model processes every input token and builds internal state for it, the **KV cache**. If the next request starts with the **same prefix** (the same system prompt, tool definitions and documents), that work can be reused instead of redone. **Prompt caching** exposes this reuse: cached input tokens are billed at a fraction of the normal price, and time-to-first-token drops, often a lot for long prompts.

## How it works

- **Prefix match only.** The cache covers the request from the first token up to a cache point. Change one token early in the prompt and everything after it is a miss.
- **Explicit or automatic.** Some APIs cache automatically above a minimum length. Others let you mark breakpoints, for example a `cache_control` marker on a content block, and charge a small premium to *write* the cache.
- **Short-lived.** Entries expire after minutes of inactivity unless they're used again, with longer lifetimes available on some APIs.
- **Exact and per-organization.** Caches aren't shared across accounts, and hits are reported in the response's usage fields.

## Structure prompts for it

Order content from **most stable to most variable**:

```text
[ tools ] [ system prompt ] [ reference docs / examples ]  ◂ cache point
[ conversation history ]                                   ◂ cache point (moves each turn)
[ the new user message ]
```

- **Keep the prefix byte-for-byte stable.** Don't put a timestamp, request ID or the user's name in the system prompt. Put variable data after the cached part.
- **Keep tool definitions in a fixed order.** Reordering them changes the prefix.
- **In multi-turn chats and agents,** move the cache point forward each turn so the growing history is reused too. That's where the biggest savings are, because every turn resends everything before it. See [the tool-use loop](https://levelcode.org/concepts/tool-use-loop).

## When it pays off

- Long system prompts or tool sets reused across many requests.
- Chatting with a large document: cache the document and vary the question.
- Agents that run many steps over the same context.
- Many [few-shot examples](https://levelcode.org/concepts/few-shot-prompting).

It does little for short, one-off prompts, or when every request has a unique prefix. Check the cache-read and cache-write token counts in production. A hit rate near zero almost always means something variable crept into the prefix.

This is different from caching *responses* in your application, which skips the model call entirely but only helps when the whole request repeats. See [caching patterns](https://levelcode.org/concepts/caching-patterns) for that.