vLLM shipped PagedAttention in 2023 and made prefix caching routine, but only within a single model: KV caches are model-bound. Swap the weights and every cached token goes cold, forcing a full re-prefill of the accumulated context. A design that relaxes this, a lightweight reader trained on the target model to interpret another model’s stored key-value state instead of recomputing it, would turn cache invalidation into an adaptation problem. Whether such a reader can be made faithful is an open empirical question, and the adjacent evidence argues for skepticism. Until per-pair transfer validation exists, production teams should keep treating KV caches as model-bound.
What do prefix caches cost you today?
Prefix caching eliminates recompute strictly within a single model: the stored KV state is a function of the weights that produced it, so swapping weights invalidates every cached token and forces a full re-prefill of the accumulated context.
The economics that make this painful are well understood. Inference splits into prefill, where the model processes the prompt and writes key-value tensors for every token, and decode, where it generates one token at a time against that state. Prefill cost grows with context length, and modern workloads carry long contexts: multi-turn dialog histories, retrieved documents, agent tool traces. Prefix caching exists because re-prefilling an unchanged prompt on every turn is waste. The serving stack that operationalized this is vLLM, the open-source inference framework originally built at UC Berkeley’s Sky Computing Lab, whose core contribution is PagedAttention, a memory-management scheme that stores KV caches in pages so they can be shared and reused across requests with the same prefix.
The catch is baked into the design. PagedAttention shares KV pages between requests hitting the same model. The cache is not a portable artifact; it is intermediate state keyed to a specific weight set. Change the weights and the cached tensors stop meaning anything. Every operation that changes weights therefore carries a hidden recompute bill:
- Model upgrades. Promote a fine-tuned checkpoint or a new version and every warm prefix in the fleet goes cold at once.
- A/B routing. Split traffic between two candidate models and each request must prefill against whichever model it lands on; there is no shared warm state.
- Provider failover. Move a session to a backup endpoint running a different model and the full conversation history gets re-prefilled from token zero.
This assumption is now expensive enough that it has attracted venture capital. In January 2026, TechCrunch reported that vLLM’s creators launched a startup, Inferact, to commercialize the project, raising $150 million in seed funding.1 That figure is a secondhand report and should be treated as such, but the direction is clear: serving economics is a market now, and the model-bound cache is one of its bigger line items.
What would a cross-model reader have to do?
The design is easy to state: a small trained reader attached to the target model consumes another model’s stored KV state and lets the target continue from it, converting what is currently a cache-invalidation event into an adaptation problem.
The conceptual logic is defensible. Two transformers do not produce aligned KV spaces a priori; the key and value geometry of model A’s layer 18 means nothing to model B’s layer 18. But if the two models were trained on overlapping data distributions, their internal representations may be related by a learnable mapping, and a lightweight target-side module could in principle learn that mapping offline, per model pair. If that holds, the operational sequence changes from “invalidate and re-prefill” to “translate and resume.”
The evidentiary situation needs stating plainly. Nothing in the available evidence demonstrates the mapping on any model pair: no per-pair fidelity measurements, no cross-architecture results covering mismatched tokenizers or dense-versus-MoE pairs. A missing number is preferable to an invented one, so none is quoted here.
That leaves the idea in a specific epistemic tier: a plausible mechanism, no demonstrated gains. That tier is exactly where a decision framework earns its keep, because the question for a serving team is not “is the idea right” but “what would have to be true, per model pair, before I route production traffic through transferred state.”
How stable is cached state even inside one model?
Even within a single model, the usefulness of cached KV state swings substantially with management policy: the Fractional Decay KV-Cache paper (arXiv:2608.18098) reports FD-KVC beating H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, while adapting to new topics 3.6x faster.2
The study details matter for the argument. The evaluation ran across five multi-turn dialog scenarios with 600 dialogs each,2 and FD-KVC also posted the highest topic diversity of all tested methods at 80.6%. These are not marginal deltas from a toy setup; they show that two reasonable policies for deciding which cached tokens to keep produce measurably different late-turn behavior in the same model with the same weights.
Carry that result one step sideways. If swapping the eviction policy inside one model moves late-turn alignment by nearly seven points, then the fidelity of KV state that has been translated between models cannot be assumed. It has to be measured, per pair, per workload. The within-model evidence is the strongest available lower bound on how fragile cached state is, and it points the wrong way for anyone hoping transferred state is faithful by default.
When would adapting a reader beat recompute?
Reader adaptation pays off only when the cost of re-prefilling accumulated context exceeds the combined cost of training and validating a per-pair reader, and nothing in the available evidence demonstrates that inequality for any model pair.
The decision decomposes into five axes, and each one is checkable before any reader code gets written:
- Recompute cost versus adaptation cost per pair. Recompute cost is a function of prefix length, concurrency, and how often cache-invalidation events occur (deploys, routing flips, failovers). Adaptation cost is the reader training run plus the validation harness, amortized over the lifetime of the pair. Pairs that live for months with frequent invalidations favor adaptation; pairs that exist for a one-week A/B test do not.
- Tokenizer and architecture compatibility. A reader can only translate state if the token streams align. Matched tokenizers are a precondition the available evidence does not show being relaxed.
- Workload shape. The FD-KVC numbers show cache utility concentrates differently in multi-turn dialog than in single-shot prompts. Long multi-turn sessions have the most to gain from transfer and, per the late-turn alignment result, the most to lose from degraded state.
- Cache management policy sensitivity. Whatever eviction and relevancy policy the serving stack applies will interact with transferred state in ways that need their own measurement.
- Serving-stack coupling. The failover path runs through the serving engine. In a vLLM/PagedAttention deployment, transferred state still has to be paged, evicted, and scheduled like native state; the reader adds a translation step, not an exemption from memory management.
| Scenario | Tokenizer/architecture match | Recompute exposure | Reader validation burden | Default today |
|---|---|---|---|---|
| Version upgrade within one model family | Likely matched | Moderate: all warm prefixes go cold at once | Lowest of the four; one direction, one pair | Budget full recompute; pilot a reader only behind a fidelity gate |
| A/B routing between two deployed models | Matched only if same tokenizer family | High if routing flips mid-session | Continuous, and both directions need separate readers | Keep per-model caches; share only the tokenized prompt |
| Provider failover to a different vendor stack | Often mismatched | Full context, at the worst possible moment | Highest, and currently unverifiable | Full recompute; warm the backup in shadow if latency matters |
| Cross-architecture pair (dense to MoE) | Mismatched by construction | Total | Unbounded; transfer unverified in available evidence | Do not attempt outside a research sandbox |
Where does transfer fidelity degrade?
The known degradation axes are tokenizer mismatch, architecture mismatch between dense and mixture-of-experts models, and long-dialog drift, and adjacent evidence says model families differ enough that none of these can be waved through without per-pair measurement.
On architecture differences, the density-matrix study of fine-tuning transitions (arXiv:2606.07559) examined behavior across five transformer architectures from two families spanning a sixfold parameter range, on ten contexts whose correct and competing completions share substantial embedding overlap. The design exists precisely because families do not behave interchangeably; representation-level similarity at the embedding layer does not guarantee aligned behavior deeper in the stack. A reader trained on one family’s KV geometry has no guarantee of transferring to another family’s, which is why the cross-architecture case is the weakest plank in any such proposal.
On capability specificity, the empirical analysis of AI post-training (arXiv:2608.19072) found that an experience-driven scaffold improved execution by +12.6 points on GSM8K and +40.8 on HumanEval while leaving strategy static. Adaptation gains, in other words, arrive unevenly across capabilities rather than uniformly. Expect the same shape from KV transfer: a reader might preserve retrieval fidelity for extractive tasks while degrading multi-step reasoning that depends on subtler attention patterns. A single aggregate fidelity score will hide exactly the failures you care about.
On dialog drift, the FD-KVC result already cited is the warning. Late-turn alignment is where within-model cache policies diverge most, and late turns are where transferred state will have drifted furthest from what a native prefill would have produced. Any fidelity gate that samples only early turns will pass state that fails in production.
What validation gates belong in front of transferred state?
A per-pair fidelity gate measured on your own multi-turn traffic is the price of admission for any reader-based transfer, and small gate models making pre-inference decisions are already an established pattern, not a novel burden.
The precedent is the Document-Reasoning Balancer work (arXiv:2608.18591), which trained DRB, an approximately 1B-parameter estimator combining SigLIP-2 and Qwen3-0.6B, to predict ordinal model performance across budget levels at 0.753 weighted F1. The relevant lesson is not the score; it is that a model roughly two orders of magnitude smaller than the systems it gates can carry a real routing decision. A transfer-fidelity gate fits the same shape: a cheap estimator or a replay harness that decides, per pair and per workload class, whether transferred state clears the bar before production traffic flows through it.
A concrete checklist for any team evaluating reader-based transfer:
- Tokenizer identity check. Confirm identical vocabularies and merge rules for the pair. A mismatch here ends the evaluation; it does not get a workaround.
- Shadow-mode replay. Run production dialog transcripts through both paths, native recompute and transferred state, and compare outputs turn by turn rather than on final answers only.
- Late-turn weighting. Weight the fidelity metric toward later turns, per the FD-KVC versus H2O evidence that relevancy diverges as dialogs lengthen.
- Cache-policy audit. Fix the eviction and relevancy policy before evaluating transfer, or you will attribute policy-induced degradation to the reader and vice versa.
- Revalidation on weight changes. Any checkpoint update on either side of the pair invalidates the previous gate result. Pair versioning belongs in the deploy pipeline, not in a wiki page.
- Rollback to recompute. The fallback path must remain warm. A reader that fails open to stale state is worse than no reader.
Should you plan for portable KV caches?
No: treat KV caches as model-bound in production today, budget full recompute for upgrades, A/B routing switches, and provider failover, and revisit reader-based transfer only for matched-tokenizer pairs once a fidelity gate on your own multi-turn traffic passes.
That is the conservative reading, and it is the correct one given the evidence state. Nothing in the available research demonstrates faithful cross-model transfer on any pair. The adjacent evidence cuts against optimism: within one model, cache utility is policy-sensitive enough to swing late-turn alignment by 6.7 points; across families, behavior diverges over a sixfold parameter range; and adaptation gains arrive capability by capability, not uniformly.
What would change the verdict is specific: per-pair fidelity data on matched-tokenizer pairs first, then on mismatched-tokenizer and dense-to-MoE pairs. If that appears, the decision table above gets its first transfer evidence and the “default today” column can start moving.
Until then, the model-bound assumption stands. It has held because cached state is easy to store and hard to trust, and the burden sits where it belongs: anyone claiming portable KV state gets to prove it, pair by pair, on someone else’s traffic.
Frequently Asked Questions
Does the cross-model KV transfer claim apply to Mixture-of-Experts (MoE) architectures?
No, the available evidence does not cover dense-to-MoE pairs. The research brief explicitly flags cross-architecture transfer as unverified, and the density-matrix study cited in the article shows that representation-level similarity does not guarantee aligned behavior across different transformer families, making MoE transfer a high-risk, unproven scenario.
How does the cost of a fidelity gate compare to the cost of full recompute?
A fidelity gate is modeled on the Document-Reasoning Balancer (DRB), which uses a ~1B parameter estimator to make routing decisions. This is significantly cheaper than full recompute for long contexts, but the gate itself requires per-pair validation on your own traffic. The cost floor is the training and validation of this small estimator, which must be amortized over the lifetime of the model pair to justify the adaptation over simple re-prefill.
What specific metric should teams use to validate transferred KV state?
Teams should weight fidelity metrics toward late-turn alignment, as the FD-KVC vs. H2O comparison shows a 6.7% divergence in this area. Sampling only early turns will pass state that fails in production, so the validation loop must compare outputs turn-by-turn on multi-turn dialog transcripts rather than relying on final answer accuracy or aggregate scores.
Is the $150 million seed funding for Inferact a verified fact?
No, the figure is a secondhand report from TechCrunch via Wikipedia and lacks primary source verification in the current corpus. While it indicates market interest in serving economics, the specific amount and the startup’s current status should be treated as unconfirmed until verified against primary financial disclosures or official announcements from the vLLM creators.
