The KV Cache: Why Your API Bill Is Higher Than It Should Be

If you’ve used the Claude or OpenAI APIs at any meaningful scale and never touched prompt caching, you’ve been overpaying. Potentially by 90%. Not as a rounding error — as a structural tax on not understanding what’s happening under the hood.

This article explains the KV cache: what it is, why it exists, why it’s the reason input tokens cost less than output tokens, and how prompt caching exposes it as a first-class cost lever. By the end you’ll have both the mental model and the practical knowledge to cut your API bills significantly.

Why Generating Text Is Expensive

To understand the KV cache, you need to understand the core inefficiency it solves.

LLMs are autoregressive — they generate one token at a time, and each new token is predicted by attending to every token that came before it. So when the model generates token number 500, it has to run attention computations over tokens 1 through 499 to decide what comes next. When it generates token 501, it repeats that process over tokens 1 through 500. Every generation step, the full history gets reprocessed.

The attention computation at the heart of a transformer takes every token’s representation, projects it into three vectors — Query (Q), Key (K), and Value (V) — and then computes how much each token should attend to every other token via a dot product between Q and K, weighted by V. This operation is O(n²) in sequence length: double the context length, quadruple the compute.

Here’s the redundancy: the Key and Value vectors for all the tokens that already exist in the sequence don’t change between generation steps. Token 47’s K and V vectors were computed when token 47 was first processed. They’re the same at step 200, step 300, step 499. Recomputing them at every generation step is pure waste.

The KV cache eliminates that waste. After the initial prompt is processed (the “prefill” phase), the K and V vectors for every token in the prompt are stored in GPU memory. During generation (the “decode” phase), the model only computes K and V for the single new token being generated, then retrieves the cached K and V for everything else. This transforms the attention computation from O(n²) to effectively O(n) per generation step.

The speedup is significant. The memory cost is also significant — for a model like LLaMA-2 13B, the KV cache consumes roughly 1 MB per output token. Over a 4K token context, that’s ~4 GB per sequence, roughly comparable to the model weights themselves. At 200K tokens, the math gets uncomfortable fast, which is why long-context serving is expensive in ways that don’t show up in the per-token pricing directly.

Why This Explains Input vs. Output Token Pricing

You’ve probably noticed that output tokens cost 3–5x more than input tokens across every major provider. Most explanations hand-wave this as “generation is harder.” The KV cache is the precise reason.

Input tokens (your prompt) go through the prefill phase. The entire prompt is processed in a single forward pass — it’s parallelizable, fast, and computationally efficient. The K and V vectors for all prompt tokens get computed at once and stored.

Output tokens go through the decode phase. Each token is generated sequentially — you cannot parallelize autoregressive generation, because token N depends on token N-1. Each step requires a full attention operation that reads from the KV cache and writes new K and V entries. The memory bandwidth and sequential compute required per output token are substantially higher than per input token.

This is also why Time to First Token (TTFT) is a separate, important metric from total generation latency. TTFT is dominated by the prefill phase — processing your entire prompt. Total latency is dominated by the decode phase — how many tokens you’re generating and at what speed. A 100K token prompt with a 50 token response will have a long TTFT but fast total generation. A short prompt with a 2000 token response will have near-instant TTFT but slow total generation. Designing prompts with awareness of this split is part of building responsive AI UX.

Prompt Caching: The KV Cache as a Product Feature

Everything above happens automatically inside every LLM inference engine. Prompt caching is what happens when providers expose that internal KV cache as a billable API feature — letting the same K and V vectors be reused across separate API requests, not just within a single request.

The concept: if you’re making many API calls that share a large common prefix — a system prompt, a document, a codebase — the model would normally recompute the K and V vectors for that shared prefix on every single call. With prompt caching, those vectors are stored server-side and reused. You pay to write the cache once. Every subsequent read is a fraction of the cost.

Anthropic’s implementation (Claude):

Cache write (5-minute TTL): 1.25x the base input token price — a 25% premium to populate the cache
Cache read: 0.1x the base input token price — a 90% discount on all subsequent reads
1-hour TTL option: 2x the base price for the write, same 90% read discount
Minimum cacheable block: 1,024 tokens
Matching is exact — one changed character invalidates the cache entirely

OpenAI’s implementation:

Automatic — no explicit API changes required, the system detects repeated prefixes automatically
50% discount on cache hits (vs. Anthropic’s 90%)
1-hour TTL (vs. Anthropic’s 5-minute default)
Same exact-match requirement

The tradeoff between the two is clear: Anthropic offers a deeper discount but requires explicit cache_control markers in your API calls and has a shorter default TTL. OpenAI is zero-config but halves the savings. For high-volume applications where the same large context is queried repeatedly in short windows, Anthropic’s 90% discount compounds quickly into real money.

The Numbers Are Not Abstract

A concrete scenario: you’re building a compliance Q&A bot (familiar territory if you’re in fintech) backed by a 100,000-token policy document. Users ask questions about the document throughout the day.

Without caching: every API call sends the full 100,000-token document as input. At Claude Sonnet’s input pricing, that’s $0.30 per call just for the document, before the user’s question or the model’s response.

With caching: the first call of the day writes the cache at 1.25x cost — $0.375. Every subsequent call reads from cache at 0.1x — $0.03 per call. At 100 calls per day, you go from $30/day to $3.375/day. That’s a real number: roughly $800/month saved on a single document-heavy app running at modest volume.

One developer reported dropping from $8,000/month to $800/month on a RAG system by implementing caching. The 90% figure isn’t marketing — it’s accurate when your workload fits the pattern.

When it doesn’t work in your favor: caching has a write premium. If a cached block is only read once before the TTL expires, you paid 1.25x for zero benefit. Low-volume applications or workloads where context changes every call won’t benefit. The break-even is roughly 2 reads per write within the TTL window — beyond that, you’re saving money.

What’s Actually Worth Caching

Not everything in a prompt is equally good to cache. The principle is: cache content that is large, static across multiple requests, and positioned at the beginning of the prompt (since caching requires prefix matching).

High-value cache candidates:

System prompts with detailed instructions — if your system prompt is 2,000+ tokens of role definitions, output constraints, and behavioral rules, cache it. It changes rarely and goes on every call.
Documents for Q&A or analysis — any workflow where users ask multiple questions about the same document. Books, legal contracts, codebases, policy documents. The document is static; the questions change. Cache the document.
Few-shot example sets — if you’re using many-shot prompting with 20+ examples to shape model behavior, those examples are identical across every call. Cache them.
Tool/function definitions — if you’re passing a large set of tool schemas on every agentic call, those schemas are static. Cache them.
Conversation history in multi-turn apps — the earlier turns of a conversation don’t change. As the conversation grows, the cached prefix grows with it. Anthropic’s automatic caching moves the breakpoint forward as the conversation extends.

What not to cache: the user’s actual question, any dynamic context that changes per-request, or anything under the 1,024-token minimum. Trying to cache short content wastes the write premium.

Implementation in Practice

The API change for Claude is minimal — add cache_control to the content block you want cached:

response = client.messages.create(\n    model="claude-sonnet-4-5",\n    max_tokens=1024,\n    system=[\n        {\n            "type": "text",\n            "text": "You are a compliance assistant...",  # short, don't cache\n        },\n        {\n            "type": "text",\n            "text": full_policy_document,               # 100K tokens, cache this\n            "cache_control": {"type": "ephemeral"}\n        }\n    ],\n    messages=[{"role": "user", "content": user_question}]\n)

The response includes cache_creation_input_tokens on the first call and cache_read_input_tokens on subsequent hits. Log these. If you’re not seeing cache reads where you expect them, debug your content structure — exact prefix matching means even whitespace differences will miss the cache.

For OpenAI, caching is automatic — no code changes required. You can verify hits by checking the cached_tokens field in the usage response object.

The Architecture Implication Nobody Mentions

Prompt caching changes what’s economically viable to build. Before caching, every token in every request cost the same. This forced a tradeoff: longer context = better answers, but unacceptable cost at scale. Developers compensated by aggressively truncating context, optimizing prompts to be short, and building more complex retrieval systems to avoid sending full documents.

With caching, a 100K-token context that’s read 100 times costs roughly the same as sending that context 10 times without caching. The per-read cost collapses. Architectures that were previously cost-prohibitive — full codebase context for every code question, entire policy manuals in every compliance query, long conversation histories in every multi-turn call — become viable.

This is not a small optimization. It’s a shift in what the cost function looks like for context-heavy applications. If you’re building anything where users interact repeatedly with the same large context — document Q&A, code analysis, knowledge base assistants, agentic systems with stable tool sets — prompt caching should be in your architecture from day one, not retrofitted later.

The Summary

The KV cache is the inference optimization that makes token generation practical at all — it eliminates the quadratic recomputation of attention states across generation steps. It’s why input tokens are cheaper than output tokens: prefill is parallelizable, decode is not.

Prompt caching exposes the server-side KV cache as an API feature, letting you amortize the compute cost of a large shared context across many requests. Anthropic’s implementation offers a 90% read discount with explicit cache_control markers. OpenAI’s is automatic at a 50% discount. Both have exact-match requirements and TTL constraints that determine when they’re worth using.

The break-even is simple: if your cached prefix is read more than twice per write within the TTL window, you’re saving money. At typical document Q&A volumes, the savings compound fast.

Most developers building production AI apps haven’t implemented this. The ones who have are quietly running at a fraction of the cost of the ones who haven’t.

Under The Hood covers the mechanics that the AI product ecosystem hand-waves.