# Cost & latency

> Estimate LLM costs, cut them with prompt caching, batching and model routing, and make responses feel fast with streaming.

AI Engineer roadmap · Agents & production · 18 min read · https://levelcode.org/roadmaps/ai-engineer/cost-and-latency

An LLM feature that works in a demo can still fail in production because it costs $40,000 a month or makes users wait eight seconds for the first word. Both problems are predictable and mostly fixable. This lesson shows you how to estimate cost per request, then works through the main levers in order of how much they usually save without hurting quality: prompt caching, batch processing, choosing models and effort per task, and streaming plus a few latency techniques that make an app feel fast.

## Where the money goes

You pay per token, with separate prices for input and output. Prices as of October 2026, per million tokens (MTok):

| Model | Input | Output | Typical use |
| --- | --- | --- | --- |
| `claude-opus-5-5` | $4.00 | $20.00 | Hardest reasoning, long agentic tasks |
| `claude-sonnet-5-5` | $2.00 | $10.00 | Strong default for production features |
| `claude-haiku-5-5` | $0.10 | $0.50 | High-volume classification, extraction, routing (prompts up to 100K tokens) |

Output tokens cost five times as much as input tokens, and they include the model's thinking: current models think adaptively by default, and those tokens are billed as output.

A worked example. A support assistant on Sonnet 5.5 sends a 4,000-token system prompt with tool definitions, 2,500 tokens of retrieved documents and a 500-token conversation, and gets back 500 tokens:

```text
input   7,000 tokens x $2.00 / 1M  = $0.0140
output    500 tokens x $10.00 / 1M = $0.0050
per request                        = $0.0190
x 50,000 requests per day          = $950 per day
```

Two habits keep you honest. **Measure**, don't guess: every response carries a `usage` object; log it with the feature name on every request. And **estimate before sending** when inputs vary widely:

```python
import anthropic

client = anthropic.Anthropic()

count = client.messages.count_tokens(
    model="claude-sonnet-5-5",
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": long_document}],
)
print(count.input_tokens)  # reject or truncate before paying for a huge request
```

Think in **cost per completed task**, not per token. A cheaper model that fails half the time and triggers a retry, or a human escalation, is not cheaper.

## Prompt caching

In the example above, the 4,000-token system prompt is identical on every request, yet you pay full price to process it 50,000 times a day. **Prompt caching** stores the processed prefix of a prompt so that later requests starting with the same bytes read it from cache instead.

The rules follow from one fact: **caching is a prefix match.** The API renders `tools`, then `system`, then `messages`, and a `cache_control` marker caches everything up to and including the block it sits on. Any byte that differs earlier in the prompt means a miss for everything after it.

```python
SYSTEM = [
    {
        "type": "text",
        "text": SUPPORT_INSTRUCTIONS,           # long, identical for every request
        "cache_control": {"type": "ephemeral"},  # cache tools + system up to here
    }
]

def ask(question: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=4096,
        tools=TOOLS,        # rendered first, so it is inside the cached prefix
        system=SYSTEM,
        messages=[{"role": "user", "content": question}],  # varies, after the marker
    )
    u = response.usage
    print(f"uncached={u.input_tokens} written={u.cache_creation_input_tokens} read={u.cache_read_input_tokens}")
    return "".join(b.text for b in response.content if b.type == "text")

ask("How do I change my plan?")
ask("Can I get an invoice in euros?")
```

```text
uncached=14 written=4012 read=0
uncached=17 written=0 read=4012
```

The first call writes the cache, the second reads it. The economics: a 5-minute cache write costs 1.25x the normal input price (2x for the optional 1-hour TTL, `{"type": "ephemeral", "ttl": "1h"}`), and a read costs about a tenth of it (on Opus 5.5, $0.20/MTok, a twentieth). Every read also refreshes the 5-minute timer, so steady traffic keeps the cache warm indefinitely. Applying that to the support assistant:

```text
cached prefix  4,000 x $0.20 / 1M = $0.0008   (was $0.0080)
rest of input  3,000 x $2.00 / 1M = $0.0060
output           500 x $10.00 / 1M = $0.0050
per request                       = $0.0118   (38% less: $590/day instead of $950)
```

For multi-turn chats, add top-level automatic caching, `cache_control={"type": "ephemeral"}` as an argument to `messages.create`, which moves the breakpoint forward as the conversation grows so each turn reads all previous turns from cache.

> [!WARNING]
> Caching fails silently: requests still succeed, the bill is just higher. Common causes are a timestamp or user name interpolated into the system prompt, `json.dumps` without `sort_keys=True` in prompt content, tool lists that differ per user, or switching models (caches are per model). Prefixes shorter than the model's minimum (512 tokens on current models) are never cached. Check `cache_read_input_tokens` in production, and again after every change to how prompts are assembled.

Total input is `input_tokens + cache_creation_input_tokens + cache_read_input_tokens`; `input_tokens` alone is only the uncached remainder.

## Batch processing

Much LLM work has nobody waiting for it: nightly summaries, backfilling tags on a million records, eval runs. The **Message Batches API** processes requests asynchronously at **50% of the normal price** on every token, and the discount stacks with caching. A batch holds up to 100,000 requests (or 256 MB); most finish within an hour, and every batch completes or expires within 24 hours.

```python
import time
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

batch = client.messages.batches.create(
    requests=[
        Request(
            custom_id=f"ticket-{t['id']}",
            params=MessageCreateParamsNonStreaming(
                model="claude-sonnet-5-5",
                max_tokens=2048,
                system=SYSTEM,  # same cached instructions as the interactive path
                messages=[{"role": "user", "content": t["body"]}],
            ),
        )
        for t in tickets
    ]
)

while (batch := client.messages.batches.retrieve(batch.id)).processing_status != "ended":
    time.sleep(60)

for result in client.messages.batches.results(batch.id):
    if result.result.type == "succeeded":
        msg = result.result.message
        save_summary(result.custom_id, "".join(b.text for b in msg.content if b.type == "text"))
    else:  # errored, canceled or expired: collect for resubmission
        retry_later(result.custom_id, result.result.type)
```

Results can come back in any order, which is why every request carries a `custom_id`. Batches are single-shot: no tool loop inside a batch request, and no streaming.

> [!TIP]
> Eval runs are an ideal batch workload: hundreds of independent requests, and nobody minds waiting an hour. Moving your eval suite to batches halves its cost, which makes it cheaper to run it on every prompt change.

## Choosing models and effort per task

The biggest price differences are between models: Haiku 5.5 is 20 times cheaper than Sonnet 5.5 per token. But only your evals can tell you whether a cheaper model is good enough for a given task. A sound process:

1. Get the feature working on a strong model and build an eval for it.
2. Try lower **effort** on the same model. `output_config={"effort": "low"}` (levels `low`, `medium`, `high`, `xhigh`, `max`) makes the model think less and answer more tersely, cutting output tokens and latency. On many tasks `medium` scores the same as `high`.
3. Try the next model tier down and rerun the eval. Keep it only if quality holds.

When requests vary in difficulty, **route** them: a cheap model classifies, and you pick the model per request.

```python
def pick_model(question: str) -> str:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        output_config={"effort": "low"},
        system="Label the request. Reply with one word. simple: factual lookups, greetings, "
               "status questions. complex: multi-step reasoning, troubleshooting, anything unclear.",
        messages=[{"role": "user", "content": question}],
    )
    label = "".join(b.text for b in response.content if b.type == "text").strip().lower()
    return "claude-haiku-5-5" if label == "simple" else "claude-sonnet-5-5"
```

Note the default: anything that is not clearly `simple` goes to the stronger model. For a bounded label like this one, a [decision model](https://levelcode.org/concepts/decision-models) such as Jev or Clef is a cheaper, faster classifier: it returns a probability for each label instead of text, so "not clearly simple" becomes a threshold you can tune. A second pattern is **escalation**: try the cheap path first and retry on a stronger model only when a check fails (output does not validate, tests fail, the answer cites nothing). Keep one model per conversation, though; switching models mid-conversation discards the prompt cache.

`max_tokens` is a safety cap, not a cost control. The model does not see it; hitting it just cuts the answer off (`stop_reason == "max_tokens"`), and you pay for a useless partial response. To get shorter answers, ask for a specific format and length in the prompt.

## Latency and streaming UX

Two numbers describe latency: **time to first token** (TTFT), how long before anything appears, and **output speed**, tokens per second after that. Users judge mostly by TTFT. Streaming does not make generation faster, but it shows progress immediately instead of after the whole answer is done.

```python
import time

start = time.perf_counter()
first = None
with client.messages.stream(
    model="claude-sonnet-5-5",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Explain our refund policy in three sentences."}],
) as stream:
    for text in stream.text_stream:
        if first is None:
            first = time.perf_counter() - start
        print(text, end="", flush=True)
    final = stream.get_final_message()

print(f"\nTTFT {first:.2f}s, total {time.perf_counter() - start:.2f}s, "
      f"{final.usage.output_tokens} output tokens, stop_reason={final.stop_reason}")
```

Because the model may think before writing visible text, TTFT for text includes thinking time; lower effort shortens it. Other levers, roughly from biggest effect:

- **Generate less.** Output speed is the bottleneck for long answers; a 150-word answer arrives in a third of the time of a 450-word one.
- **Use a smaller model or lower effort** for steps where quality allows.
- **Cache long prefixes.** A cache hit skips reprocessing those tokens and reduces TTFT on long prompts.
- **Parallelise independent calls** with `anthropic.AsyncAnthropic` and `asyncio.gather` instead of awaiting them one after another.
- **Remove sequential round trips.** A three-step chain costs three TTFTs; merge steps that do not need separate calls.

On the interface side: render streamed text as it arrives, show which tool an agent is running ("Searching orders..."), offer a stop button that closes the stream (you stop paying for tokens not yet generated), and keep the input usable while an answer streams.

## Practice

A nightly job summarises 20,000 support tickets with Sonnet 5.5 using standard synchronous calls. Each request has 1,500 tokens of shared instructions and examples, an average 800-token ticket, and a 200-token summary.

1. Compute the current nightly cost.
2. Compute the cost after moving to the Batches API.
3. Compute the cost with batching plus caching of the shared instructions (assume every request reads the cache at the batch rate of half of $0.20/MTok; ignore the few cache writes).
4. Compute the standard and batch costs on Haiku 5.5 (no caching), and describe what you would need before switching.

**Solution**

```text
1. Standard, Sonnet 5.5
   input  2,300 x $2.00 / 1M  = $0.00460
   output   200 x $10.00 / 1M = $0.00200
   per ticket $0.00660 x 20,000 = $132.00 per night

2. Batches API (50% off everything)
   $132.00 / 2 = $66.00 per night

3. Batch + cached instructions
   cached  1,500 x $0.10 / 1M = $0.00015
   ticket    800 x $1.00 / 1M = $0.00080
   output    200 x $5.00 / 1M = $0.00100
   per ticket $0.00195 x 20,000 = $39.00 per night   (70% less than step 1)

4. Haiku 5.5
   standard: 2,300 x $0.10 / 1M + 200 x $0.50 / 1M = $0.00033 x 20,000 = $6.60
   batch:    $3.30 per night
```

Steps 2 and 3 do not change the model's output, so they are safe wins: nobody reads the summaries until morning, and the instructions are identical for every ticket. Cache hits inside a batch are best-effort because requests run concurrently, so treat step 3 as the optimistic case and confirm from `usage` in the results.

Step 4 is a quality trade-off. Before switching, build an eval of 50 or more real tickets with summaries graded by a rubric (does it state the problem, the product and the requested action?), run both models, and switch only if Haiku's scores hold up. The savings are large enough to justify the work.

## Key takeaways

- Estimate cost as input tokens times input price plus output tokens times output price; output (including thinking) is the expensive side. Log `usage` on every request.
- Prompt caching is a prefix match: keep the start of the prompt byte-identical, put a `cache_control` marker at the end of the stable part, and verify with `cache_read_input_tokens`.
- The Message Batches API halves the price of anything nobody is waiting for, and stacks with caching.
- Choose models and effort per task with evals; route or escalate between tiers, and do not use `max_tokens` as a cost control.
- Stream responses to cut perceived latency, generate less, parallelise independent calls and design the UI to show progress.

## Quiz

1. Your system prompt starts with 'Current time: 2026-10-11T09:14:07Z' followed by 5,000 tokens of fixed instructions with a cache_control marker. cache_read_input_tokens is always 0. Why?

   A. The timestamp changes on every request, so the prefix never matches and every request writes a new cache entry
   B. 5,000 tokens is too long to cache
   C. Caching only works with the Batches API
   D. cache_control must be placed on the user message

   **Answer: A.** Caching matches the exact bytes from the start of the prompt. A changing value at the front invalidates everything after it. Move dynamic values after the cached prefix, for example into the user message.

2. Which workload is the best fit for the Message Batches API?

   A. A chat assistant where users wait for each reply
   B. An agent that calls tools in a loop to fix a bug
   C. Re-classifying 300,000 archived documents overnight
   D. Autocomplete suggestions while the user types

   **Answer: C.** Batches cost 50% less but complete asynchronously, within 24 hours at most, and are single-shot. Offline bulk work is ideal; interactive and tool-loop workloads are not.

3. Answers are too long and expensive. What is the right fix?

   A. Lower max_tokens until answers get shorter
   B. Switch to the Batches API
   C. Add a timestamp to the prompt
   D. Specify the desired format and length in the prompt, and consider a lower effort setting

   **Answer: D.** The model does not see max_tokens; a low cap just truncates answers mid-sentence and you still pay for them. Asking for a specific length and format shortens output, and lower effort reduces thinking tokens.

4. Why does streaming improve the user experience even though total generation time is unchanged?

   A. Streaming uses a faster model automatically
   B. Users see the first words after the time-to-first-token instead of waiting for the entire answer
   C. Streamed tokens are billed at half price
   D. Streaming disables thinking

   **Answer: B.** Perceived latency is dominated by how long the screen stays empty. Streaming shows text as it is generated, so the wait drops from the full generation time to roughly the time to first token.