Prompt Caching Slashes Input Costs Without Solving the Real Problem

Summary

• Prompt caching reduces input-token costs by 50-90% but never touches output tokens, which dominate spend at scale.
• Caching is only effective for stable, high-frequency prefixes; low hit rates reveal a prompt-structure problem, not a caching configuration issue.
• Providers require exact-match prefixes, and silent caching failures (e.g., below minimum token counts) can make the bill worse while appearing enabled.
• The real fix is to minimize stochastic calls: convert rule-based steps to deterministic logic, then cache what's left.
• For regulated enterprises, Jinba Flow builds 80% of workflows as deterministic logic, cutting per-run costs 15-60x and ensuring auditability.

Prompt caching is well documented across every major provider and will cut a meaningful slice off an input-token bill. It will not make an LLM workflow deterministic, auditable, or cheaper on the line item that dominates spend at scale: output tokens generated by a model that is still, on every single call, making a stochastic guess. The industry has converged on caching as the default cost fix because it is easy to enable and the discount is real. That is precisely why it is dangerous. It lets an engineering organization feel like it solved a cost problem when it discounted a symptom.

What prompt caching actually does

Prompt caching, also called prefix caching or context caching, reuses the key-value (KV) cache computed for a prior request's prefix, so a new request sharing that exact prefix skips recomputing those tokens and only runs prefill on the new suffix. It is a mechanical extension of an idea that already exists inside a single request. KV caching itself is intra-request: it reuses attention states during the decode stage of one generation. Prefix caching extends that same reuse across multiple requests, which is what turns a repeated system prompt or a growing chat history into a cost saving instead of a repeated expense.

The mechanism has one rule that is difficult to work around: the prefix has to match exactly. Whitespace, formatting, token order, a single character difference breaks the cache. Two prompts that mean the same thing to a human can still miss the cache entirely if they are not byte-identical up to the cache boundary. This is the detail most cost-savings pitches skip past, and it is the detail that determines whether caching actually pays for itself in production.

The cacheable surface matters for anyone deciding what to restructure. The cacheable surface is the static system prompt, tool definitions, reference documents, and the reserialized conversation history. The user's one-off question, which sits at the end of the request, is not part of that surface and is, by definition, new every time. On Amazon Bedrock this maps directly onto system prompts, system instructions, and message history, with tool fields cacheable depending on the model.

What prompt caching is not

The most common misconception is confusing prompt caching with semantic caching. Both use the word "cache" to mean different things. Semantic caching stores the full input and output text and returns a stored answer on a match, bypassing the model entirely. Prompt caching does no such thing. It never returns a stored answer. The model still runs, generation still happens, and the organization still pays for every output token the model produces. Prompt caching is a discount on the reading of the input. It is not a shortcut around the writing of the output.

That distinction is the whole argument in miniature. There are three distinct caching layers with different economics: provider prefix caching (input-side only, exact-match, model still runs on the suffix), semantic response caching (embedding similarity, bypasses the model, saves both input and output), and exact-match response caching (hash lookup, no model call at all, best for templated queries). Only the latter two touch output cost. Provider prompt caching, the layer every vendor doc leads with, is the layer that saves the least.

How the providers compare

Every major provider now ships some version of this, and the pricing mechanics are close enough across vendors that the differences are mostly about TTL and minimum block size rather than the core economics.

Anthropic's public beta launched in August 2024 and reached general availability that December; OpenAI shipped automatic caching in October 2024; Google introduced explicit context caching at I/O in May 2024 and zero-setup implicit caching for its 2.5 models in May 2025. None of this is new or experimental anymore. It is a mature, self-serve feature, which is exactly why it has become the reflexive answer to "our LLM bill is too high."

Every provider requires the developer to actively mark the cache boundary: Anthropic with cache_control, Bedrock with "cache checkpoints," OpenAI with prompt_cache_breakpoint, Gemini with explicit or implicit caching. The ordering discipline is the same everywhere: put content from most to least stable, tool definitions first, then system prompt, then reference material, then conversation history, with the live user query last. On Anthropic specifically, changing any block invalidates that block and everything downstream of it, so dynamic data has to live at the very end or the cache collapses on every turn.

When prompt caching is the right call and when it is not

Caching earns its keep on a genuinely stable, high-frequency prefix: a long system prompt reused across thousands of calls per minute, a document loaded once and queried repeatedly, a chat history that grows but never rewrites its earlier turns. The 5-minute TTL suits high-QPS workloads; the 1-hour tier, despite its higher write premium, suits medium-frequency use; OpenAI's 24-hour extended retention suits slow-burn batch agents.

It stops earning its keep the moment the "stable" prefix is not actually stable. The sales pitch and the production reality diverge here. Bedrock's cache writes cost 1.25x the standard input rate, and Anthropic's 5-minute tier breaks even at roughly 1.4 cache reads per write, meaning that below a hit rate of about 30%, the write premium costs more than the reads save. Caching is not a strictly positive lever. Turn it on against a volatile prefix and it is possible to make the bill worse, not better, while a dashboard somewhere reports that "caching is enabled" as if that were the same thing as caching working.

There is a second failure mode that vendor documentation buries in the fine print rather than the headline. Bedrock's cache checkpoints carry per-model minimum token counts: 1,024 tokens for Claude 3.7 Sonnet, 4,096 for Claude Opus 4.5, Sonnet 4.5, and Haiku 4.5. If the marked block falls under that minimum, inference succeeds and the prefix is silently not cached. No error. No warning. The request just runs at full price while the engineering team believes the cache is live. Cross-region inference compounds the risk: at times of high demand it can increase cache writes rather than reduce them. Prompt caching is marketed as a checkbox. Operating it correctly is an integration discipline with its own failure modes, and those failure modes cost real money because they are silent.

The diagnostic nobody is using correctly

The insight that gets missed is treating cache hit rate as a metric to push toward 100%. Hit rate is better understood as a canary for prompt architecture than as a KPI to optimize directly. If a prompt is genuinely static and the hit rate is still low, under roughly 60% on what should be a stable prefix, that is not a caching problem. It is a signal that dynamic content has leaked into the part of the prompt that was supposed to stay fixed.

The clearest evidence of this is a documented case rather than a theory: ProjectDiscovery raised its cache hit rate from 7% to 84%, cutting overall LLM cost 59% across 9.8 billion tokens served from cache, by relocating dynamic "working memory" out of the system prompt and into a trailing user message. Nobody adjusted a caching setting. The fix was structural: moving state out of the block that was meant to stay fixed. That 59% reduction did not come from turning caching on. It came from realizing the prompt was not actually stable, and rebuilding it so that it was.

This reframes the whole conversation. A team staring at a 12% cache hit rate does not have a caching problem to solve with more engineering effort on cache configuration. It has a prompt-design problem that caching was exposing the whole time. The counterintuitive point is that naive full-context caching can even raise latency, because a longer cached prefix expands the recompute surface if the boundary is not deliberately controlled. The "just cache everything" instinct works against the very outcome it is chasing.

What this misses even when it is done right

Grant every one of these caveats a clean fix. Assume perfect prefix ordering, correct TTL selection, hit rates above 80%, no silent checkpoint failures. The organization has just optimized the input side of a bill that was never the whole problem.

Caching discounts range from 50% to 90% on input tokens and never touch output tokens. For a workflow where the model reads a long document and writes a short answer, that is a genuine win. For a workflow where the model reads a modest prompt and writes a long, structured output, such as a contract redline, a KYC summary, or a compliance memo, the caching discount is applied to the smaller half of the bill. And every one of those output tokens is still generated by a model making a probabilistic next-token choice. The underlying problem that regulated enterprises actually care about is untouched: the call is still stochastic, the output is still not guaranteed to be the same twice, and nothing about a 90% input discount makes that output auditable.

Jinba's workflow architecture for banks and insurers was built around this distinction. A bank's compliance team does not primarily ask "how much did the API call cost." It asks whether the same document, run twice, produces the same classification, and whether that decision can be reconstructed for an examiner eighteen months later. Prompt caching has no opinion on either question. It optimizes the meter without touching the architecture that generates the meter's readings. This tension between input-side cost discounts and deterministic execution is covered in more depth in prompt optimization versus deterministic workflows for reducing token costs.

That is the design decision Jinba made differently: rather than caching the input side of a stochastic call and hoping the discount compounds, roughly 80% of a given workflow's execution path is built as deterministic, rule-based logic, with the LLM invoked only for the remaining steps that genuinely require judgment: document classification at the edges, ambiguous field extraction, free-text summarization. Banks and insurers can apply the same approach with deterministic workflows to reduce AI costs. Because the majority of steps carry zero marginal LLM cost per execution rather than a discounted one, the cost structure is not "cheaper stochastic calls," it is "fewer stochastic calls." At scale, that is the difference between paying 15-60x less per workflow run and paying a caching-optimized but still fundamentally probabilistic bill. It also means the deterministic majority of the workflow is inherently reproducible and logged, which a caching discount was never going to deliver regardless of hit rate.

The honest counter-argument

The strongest objection to all of this is straightforward: a 90% discount on a large chunk of real spend is still real money, and dismissing it because it does not solve everything is a false choice. The two are not mutually exclusive, and a team running a genuinely stable, high-QPS prefix that is not currently cached is leaving discount on the table for no reason. If the prefix is stable and the hit rate is healthy, enabling caching is close to free money, and an engineering leader who ignores it because it is "not the real fix" is making the same mistake in the opposite direction: treating a legitimate tactical win as beneath attention. Enterprises deploying Claude at scale face the same trade-off; the on-premise Claude deployment options are covered separately.

That objection holds, and it should change the framing rather than the conclusion. Caching is a correct, low-effort optimization for genuinely stable, high-volume prefixes, and any team not running it there is leaving money on the table. The failure is not in using caching. The failure is in treating the caching discount as evidence that the cost and reliability problem has been addressed, when it has only been addressed on the input side of calls that remain stochastic on the output side. A team with heavy caching and batch-API investment can feel, reasonably, that they already solved their cost problem, and that feeling is exactly what makes the deeper fix harder to sell internally. The dashboard shows the bill went down. It does not show that every one of those calls is still a coin flip on determinism.

The optimization to run first

For an engineering leader who owns total inference cost and architecture, not just the per-token line item, the sequence matters. Before reaching for caching configuration, run the diagnostic: what is the current cache hit rate on the prompts already assumed to be stable? Cost-accountable buyers use a broader set of levers, which is laid out in the two-part CFO framework for enterprise AI cost reduction. If it is healthy — above roughly 60-80% depending on TTL tier — caching is doing its job and the remaining lever is genuinely on the cost-of-output side. If it is low, that is not a caching problem to engineer around; it is a prompt-structure problem, and the fix is the ProjectDiscovery move: find the dynamic content sitting inside the supposedly-static prefix and relocate it to the end of the request.

Only after that diagnostic should the harder question get asked: of the steps in this workflow, how many actually require a stochastic model call, and how many are being routed through an LLM out of convenience rather than necessity? Document classification against a known taxonomy, field validation against a rule set, routing decisions with clear branching logic — these do not need a model guessing at an answer it could instead be told deterministically. Every step converted from a model call to rule-based logic removes both a cost line and an auditability gap in one move. That reordering, diagnose the cache, then audit the architecture, then cache what is left, is the sequence a cost-accountable buyer should run, because it is the only one that touches both halves of the bill instead of just the half that is easy to discount.

FAQ

Does prompt caching reduce output token costs? No. Every provider's caching discount applies to input tokens read from the cache; the model still generates the response from scratch on every call, and output tokens are billed at full price regardless of how the input was served.

Is prompt caching the same as a Redis-style application cache? No. A conventional key-value or Redis cache returns a stored value with no computation involved. Prompt caching still runs the model on the new portion of the request. It only skips recomputing the KV state for the unchanged prefix, so it reduces compute, not calls.

Can a low cache hit rate mean the model itself is fine and only the prompt is broken? Yes, and this is the most useful reframing available. A low hit rate on a prompt assumed to be static usually means dynamic content, such as timestamps, session state, or retrieved context, is sitting inside the block that was meant to stay fixed. That is a prompt-engineering fix, not a model or infrastructure issue.

Does enabling prompt caching guarantee lower latency? Not automatically. Naive full-context caching can raise latency if the cached prefix is long and the cache boundary is not deliberately controlled, since a wider prefix expands the recompute surface on a miss.

Should a regulated enterprise rely on prompt caching for audit readiness? No. Caching changes how a request is billed and served, not whether the underlying model call is deterministic. A cached call and an uncached call carry the same reproducibility profile. The model is still generating a stochastic output either way.