Cost & latency

An LLM feature that works in a demo can still fail in production because it costs $40,000 a month or makes users wait eight seconds for the first word. Both problems are predictable and mostly fixable. This lesson shows you how to estimate cost per request, then works through the main levers in order of how much they usually save without hurting quality: prompt caching, batch processing, choosing models and effort per task, and streaming plus a few latency techniques that make an app feel fast.

#Where the money goes

You pay per token, with separate prices for input and output. Prices as of October 2026, per million tokens (MTok):

Output tokens cost five times as much as input tokens, and they include the model's thinking: current models think adaptively by default, and those tokens are billed as output.

A worked example. A support assistant on Sonnet 5.5 sends a 4,000-token system prompt with tool definitions, 2,500 tokens of retrieved documents and a 500-token conversation, and gets back 500 tokens:

input   7,000 tokens x $2.00 / 1M  = $0.0140
output    500 tokens x $10.00 / 1M = $0.0050
per request                        = $0.0190
x 50,000 requests per day          = $950 per day

Two habits keep you honest. Measure, don't guess: every response carries a usage object; log it with the feature name on every request. And estimate before sending when inputs vary widely:

import anthropic

client = anthropic.Anthropic()

count = client.messages.count_tokens(
    model="claude-sonnet-5-5",
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": long_document}],
)
print(count.input_tokens)  # reject or truncate before paying for a huge request

Think in cost per completed task, not per token. A cheaper model that fails half the time and triggers a retry, or a human escalation, is not cheaper.

#Prompt caching

In the example above, the 4,000-token system prompt is identical on every request, yet you pay full price to process it 50,000 times a day. Prompt caching stores the processed prefix of a prompt so that later requests starting with the same bytes read it from cache instead.

The rules follow from one fact: caching is a prefix match. The API renders tools, then system, then messages, and a cache_control marker caches everything up to and including the block it sits on. Any byte that differs earlier in the prompt means a miss for everything after it.

SYSTEM = [
    {
        "type": "text",
        "text": SUPPORT_INSTRUCTIONS,           # long, identical for every request
        "cache_control": {"type": "ephemeral"},  # cache tools + system up to here
    }
]

def ask(question: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=4096,
        tools=TOOLS,        # rendered first, so it is inside the cached prefix
        system=SYSTEM,
        messages=[{"role": "user", "content": question}],  # varies, after the marker
    )
    u = response.usage
    print(f"uncached={u.input_tokens} written={u.cache_creation_input_tokens} read={u.cache_read_input_tokens}")
    return "".join(b.text for b in response.content if b.type == "text")

ask("How do I change my plan?")
ask("Can I get an invoice in euros?")

uncached=14 written=4012 read=0
uncached=17 written=0 read=4012

The first call writes the cache, the second reads it. The economics: a 5-minute cache write costs 1.25x the normal input price (2x for the optional 1-hour TTL, {"type": "ephemeral", "ttl": "1h"}), and a read costs about a tenth of it (on Opus 5.5, $0.20/MTok, a twentieth). Every read also refreshes the 5-minute timer, so steady traffic keeps the cache warm indefinitely. Applying that to the support assistant:

cached prefix  4,000 x $0.20 / 1M = $0.0008   (was $0.0080)
rest of input  3,000 x $2.00 / 1M = $0.0060
output           500 x $10.00 / 1M = $0.0050
per request                       = $0.0118   (38% less: $590/day instead of $950)

For multi-turn chats, add top-level automatic caching, cache_control={"type": "ephemeral"} as an argument to messages.create, which moves the breakpoint forward as the conversation grows so each turn reads all previous turns from cache.

Total input is input_tokens + cache_creation_input_tokens + cache_read_input_tokens; input_tokens alone is only the uncached remainder.

#Batch processing

Much LLM work has nobody waiting for it: nightly summaries, backfilling tags on a million records, eval runs. The Message Batches API processes requests asynchronously at 50% of the normal price on every token, and the discount stacks with caching. A batch holds up to 100,000 requests (or 256 MB); most finish within an hour, and every batch completes or expires within 24 hours.

import time
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

batch = client.messages.batches.create(
    requests=[
        Request(
            custom_id=f"ticket-{t['id']}",
            params=MessageCreateParamsNonStreaming(
                model="claude-sonnet-5-5",
                max_tokens=2048,
                system=SYSTEM,  # same cached instructions as the interactive path
                messages=[{"role": "user", "content": t["body"]}],
            ),
        )
        for t in tickets
    ]
)

while (batch := client.messages.batches.retrieve(batch.id)).processing_status != "ended":
    time.sleep(60)

for result in client.messages.batches.results(batch.id):
    if result.result.type == "succeeded":
        msg = result.result.message
        save_summary(result.custom_id, "".join(b.text for b in msg.content if b.type == "text"))
    else:  # errored, canceled or expired: collect for resubmission
        retry_later(result.custom_id, result.result.type)

Results can come back in any order, which is why every request carries a custom_id. Batches are single-shot: no tool loop inside a batch request, and no streaming.

#Choosing models and effort per task

The biggest price differences are between models: Haiku 5.5 is 20 times cheaper than Sonnet 5.5 per token. But only your evals can tell you whether a cheaper model is good enough for a given task. A sound process:

1. Get the feature working on a strong model and build an eval for it.
2. Try lower effort on the same model. output_config={"effort": "low"} (levels low, medium, high, xhigh, max) makes the model think less and answer more tersely, cutting output tokens and latency. On many tasks medium scores the same as high.
3. Try the next model tier down and rerun the eval. Keep it only if quality holds.

When requests vary in difficulty, route them: a cheap model classifies, and you pick the model per request.

def pick_model(question: str) -> str:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        output_config={"effort": "low"},
        system="Label the request. Reply with one word. simple: factual lookups, greetings, "
               "status questions. complex: multi-step reasoning, troubleshooting, anything unclear.",
        messages=[{"role": "user", "content": question}],
    )
    label = "".join(b.text for b in response.content if b.type == "text").strip().lower()
    return "claude-haiku-5-5" if label == "simple" else "claude-sonnet-5-5"

Note the default: anything that is not clearly simple goes to the stronger model. For a bounded label like this one, a decision model such as Jev or Clef is a cheaper, faster classifier: it returns a probability for each label instead of text, so "not clearly simple" becomes a threshold you can tune. A second pattern is escalation: try the cheap path first and retry on a stronger model only when a check fails (output does not validate, tests fail, the answer cites nothing). Keep one model per conversation, though; switching models mid-conversation discards the prompt cache.

max_tokens is a safety cap, not a cost control. The model does not see it; hitting it just cuts the answer off (stop_reason == "max_tokens"), and you pay for a useless partial response. To get shorter answers, ask for a specific format and length in the prompt.

#Latency and streaming UX

Two numbers describe latency: time to first token (TTFT), how long before anything appears, and output speed, tokens per second after that. Users judge mostly by TTFT. Streaming does not make generation faster, but it shows progress immediately instead of after the whole answer is done.

import time

start = time.perf_counter()
first = None
with client.messages.stream(
    model="claude-sonnet-5-5",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Explain our refund policy in three sentences."}],
) as stream:
    for text in stream.text_stream:
        if first is None:
            first = time.perf_counter() - start
        print(text, end="", flush=True)
    final = stream.get_final_message()

print(f"\nTTFT {first:.2f}s, total {time.perf_counter() - start:.2f}s, "
      f"{final.usage.output_tokens} output tokens, stop_reason={final.stop_reason}")

Because the model may think before writing visible text, TTFT for text includes thinking time; lower effort shortens it. Other levers, roughly from biggest effect:

• Generate less. Output speed is the bottleneck for long answers; a 150-word answer arrives in a third of the time of a 450-word one.
• Use a smaller model or lower effort for steps where quality allows.
• Cache long prefixes. A cache hit skips reprocessing those tokens and reduces TTFT on long prompts.
• Parallelise independent calls with anthropic.AsyncAnthropic and asyncio.gather instead of awaiting them one after another.
• Remove sequential round trips. A three-step chain costs three TTFTs; merge steps that do not need separate calls.

On the interface side: render streamed text as it arrives, show which tool an agent is running ("Searching orders..."), offer a stop button that closes the stream (you stop paying for tokens not yet generated), and keep the input usable while an answer streams.

#Practice

A nightly job summarises 20,000 support tickets with Sonnet 5.5 using standard synchronous calls. Each request has 1,500 tokens of shared instructions and examples, an average 800-token ticket, and a 200-token summary.

1. Compute the current nightly cost.
2. Compute the cost after moving to the Batches API.
3. Compute the cost with batching plus caching of the shared instructions (assume every request reads the cache at the batch rate of half of $0.20/MTok; ignore the few cache writes).
4. Compute the standard and batch costs on Haiku 5.5 (no caching), and describe what you would need before switching.

1. Standard, Sonnet 5.5
   input  2,300 x $2.00 / 1M  = $0.00460
   output   200 x $10.00 / 1M = $0.00200
   per ticket $0.00660 x 20,000 = $132.00 per night

2. Batches API (50% off everything)
   $132.00 / 2 = $66.00 per night

3. Batch + cached instructions
   cached  1,500 x $0.10 / 1M = $0.00015
   ticket    800 x $1.00 / 1M = $0.00080
   output    200 x $5.00 / 1M = $0.00100
   per ticket $0.00195 x 20,000 = $39.00 per night   (70% less than step 1)

4. Haiku 5.5
   standard: 2,300 x $0.10 / 1M + 200 x $0.50 / 1M = $0.00033 x 20,000 = $6.60
   batch:    $3.30 per night

#Key takeaways

• Estimate cost as input tokens times input price plus output tokens times output price; output (including thinking) is the expensive side. Log usage on every request.
• Prompt caching is a prefix match: keep the start of the prompt byte-identical, put a cache_control marker at the end of the stable part, and verify with cache_read_input_tokens.
• The Message Batches API halves the price of anything nobody is waiting for, and stacks with caching.
• Choose models and effort per task with evals; route or escalate between tiers, and do not use max_tokens as a cost control.
• Stream responses to cut perceived latency, generate less, parallelise independent calls and design the UI to show progress.