How to Verify Your LLM Agent Actually Gets the Prompt Caching Discount

Your agent’s prompt-caching discount is probably not delivering what you think it is, and nothing in your test suite will tell you. A study of 120 real-world caching defects mined from open-source agent projects found that 90% of them are silent: no exception, no failed assertion, no crash, and the model output stays correct while only the caching degrades. The bill arrives at full price anyway.

The fix is not a better test. It is an audit: log the cached-token usage fields your provider already returns on every response, compute an actual cache-hit rate, and grep your prompt-assembly code for the handful of patterns that quietly break reuse. This article lays out that procedure, the defect patterns behind it, and the cost math for what one silent miss does to an agent run. Two caveats up front, because they shape everything below: the defect evidence comes from a single GitHub-mined preprint, so prevalence in closed commercial harnesses is unknown, and the pricing figures below are that preprint’s citation of Anthropic, OpenAI, and Google documentation rather than numbers checked against the providers’ own docs. Exact discount tiers, current field names, and cache lifetimes need re-checking against OpenAI and Anthropic docs before you quote numbers internally.

Why these failures never surface in tests

Prompt caching is supposed to be a large lever. According to the mis-integration preprint’s summary of provider materials, reusing stable prefixes “can reduce cached input costs to roughly one-tenth of the standard price and decrease inference latency by up to 80%.” Treat that figure as the paper’s citation of vendor pricing, not as independently verified documentation; it is the right order of magnitude for planning, not a number for a budget line.

The failure mode is what the paper calls a mis-integration: application code that violates provider requirements for enabling, configuring, or accounting for prompt caching. Of the 120 defects the authors mined from agent projects, 108, or 90%, remain silent. The mining used provider cache markers like cache_control, cachePoint, cachedContent, and prompt_cache_key as search indicators. The paper’s own summary is blunt: “They raise no exception, no failed assertion, and no crash, and the model output stays correct while only caching degrades.” A further 86 defects (72%) are what the authors classify as silent waste, where the cache simply stops being reused with no visible symptom.

That silence is structural, not incidental. None of the eight agent benchmark papers in a pilot audit of twelve benchmark papers disclosed inference cost in any form, so a forfeited discount appears nowhere in benchmark output and only later in billing data. If your team’s definition of “the agent works” is green tests plus good eval scores, caching waste sits entirely outside that definition.

Step 1: log the usage fields on every response

Both major API families return cache telemetry in the response usage object: Anthropic-style responses carry cache-creation and cache-read token counts, and OpenAI-style responses carry a cached_tokens field. The CacheProbe audit methodology treats this metadata as ground truth: “Some providers explicitly report how many input tokens were served from cache, providing ground-truth evidence of cache utilization independent of timing signals.”

So the first audit step is mechanical: capture those fields on every agent response and store them alongside the request, per call, not aggregated. Per-call matters because agents make many calls with varying prefixes, and an average can hide a subset of calls that never hit.

There is one trap even at this step, and it is documented rather than hypothetical. The preprint describes a bug in Anthropic’s official .NET SDK: “The API returns cache-creation and cache-read token counts as optional usage fields, but the SDK deserialized these fields into nullable properties and consumed them without a null check.” When the fields were absent, accounting broke downstream. Maintainers fixed it after the authors reported it, but the pattern generalizes: if an official SDK shipped this, your harness’s usage-field parsing deserves the same scrutiny. Verify that a missing field surfaces as a warning rather than silently reading as zero, or worse, being skipped.

One limitation on the whole metadata approach: it is not universal. CacheProbe reports that “Fireworks does not report cached token metadata, so timing analysis is the only detection method available,” and timing is noisy. In that study, Groq’s timing analysis produced p = 0.002 against the authors’ own p<10⁻⁸ significance bar, with only metadata disclosure confirming the underlying behavior. If your provider omits cached-token fields, your audit options narrow to latency comparisons, which need many samples and careful controls.

Step 2: compute a hit rate against a real threshold

Logging fields is not the same as having an answer. CacheProbe supplies a falsifiable decision rule you can adapt: detection is considered positive “when more than 50% of cache-primed requests report cached tokens exceeding 90% of the expected prefix length” (CacheProbe’s detection criterion).

Translated into an agent audit, that becomes: prime the cache by running a representative workload, then check whether the majority of subsequent calls report cached-token counts covering nearly all of the stable prefix you intended to reuse. The “expected prefix length” part forces you to write down how much of your prompt should be stable: the system prompt, tool definitions, and any static context you prepend. If you cannot state that number, you do not yet know what a hit would look like, which is itself a finding.

For comparison, figures the Unified AI Gateway paper cites from vLLM and SGLang deployments show exact-prefix caching delivering 2–4× and up to 6.4× throughput gains, with production hit rates of 52.4% on LLaVA-Next-34B and 74.1% on Vicuna-33B. Your agent’s numbers will differ, but if your audit shows single-digit hit rates on a workload dominated by repeated prefixes, something in the assembly path is breaking reuse, and the next section is where to look.

The patterns to grep for

The preprint’s taxonomy puts provider compatibility and prefix stability at the majority of the 120 defects. Three patterns recur and are cheap to check.

Volatile data at the front of the prompt. Caching works on exact prefixes, so anything that changes near the start invalidates everything after it. The paper’s Figure 2 documents zylon-ai/private-gpt placing “a sub-second timestamp at the front of the system prompt,” which broke the cached prefix on every call; the fix coarsened it to day granularity. Any per-request injection into the early prefix (timestamps, request IDs, session randomness, current date to the minute) has the same effect. The rule of thumb: push volatile content as late in the prompt as the provider’s caching semantics allow, and make what remains deterministic byte-for-byte.

Dynamic or per-user system prompts. A system prompt re-rendered per user, per tenant, or per run may be a stable prefix only within a narrow slice of traffic. That is an architecture decision with a hidden tax: personalization at the prefix position forfeits cross-request reuse for everything appended after it. Sometimes that tradeoff is worth it, but it should be made knowing the price, not discovered on the invoice.

Tool-definition ordering and gateway gating. If your harness serializes tool definitions in non-deterministic order (dict iteration, set ordering, registration order that varies by startup), the prefix churns across calls. And the altimate-code case shows a subtler version: the project “attached cache_control markers only for Anthropic-native calls, using a hard-coded set of model paths,” so identical requests routed through OpenRouter- or Bedrock-style gateways were left unmarked and could not use caching at all. If your agent can run against multiple backends, grep for wherever cache markers are attached and check the condition guarding them.

These are not exotic edge cases. Confirmed mis-integration cases in the study span AutoGPT, n8n, and Zed, with 11 additional reports awaiting maintainer response at the time of writing and none rejected.

Gateways cut both ways

If your traffic passes through a gateway or router, two distinct effects matter, and conflating them leads to wrong conclusions.

The first erases your discount. Routing decisions “can discard useful state, increase time to first token (TTFT) and input cost, and create cache-induced model lock-in,” according to the Unified AI Gateway paper. If requests with identical prefixes scatter across backend instances or models, each destination warms its own cache and the rest miss. The altimate-code case above is the application-side version: markers that never reach the provider because the route did not match a hard-coded path.

The second is a security finding, not a cost one. CacheProbe found that routing through OpenRouter with default shared credentials produced cross-account cache sharing: Groq showed 100% hit rates in cross-account scenarios, Fireworks timing evidence reached p = 4.08×10⁻¹⁵, and OpenAI showed minor leakage at 4.8% of cross-account requests in the same audit. Here the cache hits; the problem is that a prefix shared across tenants means cached state from one account can serve another, an isolation concern rather than a lost discount. The audit takeaway differs per effect: for cost, verify markers survive the hop and routing keeps prefixes sticky; for isolation, check whether your gateway configuration pools credentials across accounts.

The audit checklist

What one silent miss costs a run

The compounding mechanism is straightforward. Multi-agent workflows “require repeated LLM calls across multiple steps and maintain long-horizon task contexts,” as the ProgRouter paper puts it, and each step typically re-sends the accumulated context. If the stable prefix is a large fraction of each call’s input and the cache is working, that repeated prefix bills at roughly a tenth of standard price (the preprint’s cited figure). Break the prefix and every call bills the whole context at full rate.

Run the arithmetic on a hypothetical: an agent making 50 calls, each carrying 20,000 tokens of prefix meant to be cached, plus fresh input. With reuse working at the cited discount, the repeated-prefix portion costs about as much as 100,000 standard tokens across the run. With a timestamp near the front breaking every prefix, it costs 1,000,000. The model output is identical in both runs. Nothing fails. The difference appears only in billing, and because the miss is silent, the natural response is to blame token volume rather than the assembly bug.

This is why the second-order effect matters more than any single fix: caching shifts the cost bottleneck from token count to prefix stability. Design choices that look cost-neutral, replayed trajectories, per-user system prompts, gateway fan-out across models, now carry a tax that is invisible until someone reads the usage fields. The gateway paper’s observation about cache-induced model lock-in follows from the same math: once your economics depend on a warm cache, routing away from it has a real price.

Re-verify the moving parts before you publish numbers internally

Two details about provider caching behavior are shifting faster than articles can track, and the honest thing is to flag them rather than assert them. The preprint’s reference list cites a DEV Community post reporting that Anthropic dropped prompt-cache TTL from one hour to five minutes in 2026, and an AWS announcement of a one-hour Bedrock cache TTL in January 2026 gated to Claude Sonnet, Haiku, and Opus 4.5. Both trace to those sources rather than provider documentation, so check the current provider-cited behavior before sizing call cadence around a cache lifetime.

The durable part of the audit survives whatever changes there: log the fields, set a hit-rate alert, and treat prefix stability as a cost surface. I would start with the usage-field logging, because it is the cheapest instrument and the one whose absence let an official SDK’s null-handling bug ship unnoticed. If your provider returns no cached-token metadata at all, as Fireworks does in CacheProbe’s testing, then timing analysis is what you have, and you should demand far more samples before trusting it. Either way, the burden of proof sits with the discount: until your telemetry shows the hits, assume the misses.

Frequently Asked Questions

What threshold does CacheProbe use to determine if prompt caching is working?

CacheProbe supplies a falsifiable decision rule you can adapt: detection is considered positive “when more than 50% of cache-primed requests report cached tokens exceeding 90% of the expected prefix length” (CacheProbe’s detection criterion).

What should you do if your provider does not return cached-token metadata?

If your provider omits cached-token fields, your audit options narrow to latency comparisons, which need many samples and careful controls.