arXiv:2606.24506, a paper from Beihang University posted in June 2026, attacks a memory-economics problem in multi-tenant MoE serving: when a cluster hosts many checkpoints and most of them are cold, static FFN weights squat on GPU memory and crowd out the KV-cache that active requests actually need. Its answer is to put those two memory classes in separate pools and ship hidden states across the boundary, not KV tensors.
Why does hosting many MoE checkpoints waste GPU memory?
The binding constraint is residency, not compute. Sparse mixture-of-experts models carry large FFN weight matrices that stay resident in VRAM whether or not anyone is querying them, and in a multi-model deployment the long tail of checkpoints dominates the memory budget while contributing almost nothing to throughput.
The paper quantifies this with a cumulative distribution of token consumption on OpenRouter: roughly 90% of models are cold and consume few tokens (arXiv:2606.24506, full text). Reserving worst-case KV capacity per model on that distribution is wasteful by construction. The current generation of multi-LLM engines, MuxServe and kvcached, colocate weights and KV-cache in a single monolithic GPU pool, which couples KV capacity to weight footprint (arXiv:2606.24506, full text). That coupling is the root of the symptoms operators already recognize: low per-GPU KV capacity, weak long-context support, and requests that get rejected or OOM-killed when a handful of cold models’ weights have the cards pinned flat.
There is a subtler failure too, and it is algorithmic. KV-head-limited attention variants (MLA, MQA, GQA with fewer heads than the data-parallel group size) expose only a fraction, a half or a quarter, of replicated KV capacity to any single request under cold, low-concurrency traffic (arXiv:2606.24506, full text). A data-parallel replica can look fat on a capacity spreadsheet and sit effectively unusable, because the request cannot reach the KV head it needs on the replica it landed on.
What does CrossPool actually split, and what does it leave alone?
CrossPool separates two address spaces, not the workload. A weights pool consolidates FFN weights across the cold models, and a KV-cache pool dynamically serves whichever requests are actually active (arXiv:2606.24506). Attention and its KV-cache live in the second pool; the FFN, which dominates MoE parameter count, lives in the first (arXiv:2606.24506, full text).
Because the FFN weights are the bulky resident, pooling them across many underused models is where the memory savings concentrate: the same weight memory that was reserved per model now backs the whole cold fleet, freeing VRAM for the KV-cache where the contention actually is. The KV-cache pool, no longer forced to coexist with pinned weights, can grow to the capacity that active long-context requests need.
The paper lists three mechanisms that make the split pay off rather than reshuffle the problem. A KV-cache planner and virtualizer performs offline budget and parallelism planning with virtualized paging. A layer-wise pipeline scheduler overlaps hidden-state transfers with compute across in-flight batches. Persistent kernels with control lowering cut the CPU-to-GPU control overhead that eats throughput in small-batch regimes (arXiv:2606.24506, full text). The first two make the pools cooperate; the third ensures you do not pay a synchronization tax for the privilege of having split them.
Hidden states, not KV-cache: why isn’t this prefill-decode disaggregation?
The boundary CrossPool draws is inside the transformer layer, between attention and the FFN, so what crosses between pools is a hidden-state tensor, not a KV-cache tensor (arXiv:2606.24506, full text). Attention stays local to the KV-cache pool because that is where the keys and values it reads live. The FFN executes in the weights pool because that is where the weight matrices it multiplies live. Each pool does the half of the layer it is positioned to do without dragging the other pool’s memory across the wire.
This is a different cut from the disaggregation pattern production serving stacks have been converging on. Prefill-decode disaggregation, as implemented in NVIDIA Dynamo and examined in Groundy’s piece on prefill-decode disaggregation, splits the request lifecycle across phases and transfers KV-cache between them. CrossPool splits a layer across memory classes and transfers hidden states. Same family of idea, separating the things with different lifetimes and footprints, but a different axis. The two are compatible rather than competing; a system could disaggregate both ways. CrossPool’s own evaluation is direct evidence for that: the authors run it and both baselines in prefill-decode disaggregation mode, with prefill handled by separate temporal-multiplexing engines, and report decode-side results (arXiv:2606.24506, full text).
How real is the 10.4× time-between-tokens claim?
The 10.4× is real but narrow: it is an “up to” figure measured against a kvcached-based multi-LLM serving system, and the paper itself concedes only competitive performance on balanced workloads (arXiv:2606.24506).
The full text locates the headline precisely. The 10.4× is DeepSeek-V2-Lite’s P99 reduction at 0.8 requests per second per model on a five-A100 testbed; at the same rate Qwen3-30B-A3B and GLM-4.7-Flash showed 7.6× and 7.3×, and at 1.0 RPS the reductions fell to 2.1× for Qwen3-30B-A3B, 5.0× for DeepSeek-V2-Lite and 2.2× for GLM-4.7-Flash (arXiv:2606.24506, full text). The mechanism is tail contention: the authors attribute kvcached’s higher P99 to uncoordinated multi-model concurrency causing SM, memory-bandwidth and KV-cache contention during decode. And the static-partition baseline, which gives each model a fixed isolated placement, posted the lowest TBT of the three, at the cost of exactly the long-context pooling CrossPool exists to provide. Read that way, the number is a pooling-versus-isolation result measured against a contended baseline, not a uniform speedup.
Two things limit how far that number travels. The large gains concentrate in the cold, long-context, low-concurrency regime that motivated the design in the first place; a deployment whose traffic is hot and evenly distributed across models should expect the “competitive” result, not the headline (arXiv:2606.24506, full text). The baseline matters too. The comparison is against kvcached specifically, not against a production disaggregated stack with RDMA and a tuned transfer path. kvcached inherits the monolithic-pool coupling CrossPool is built to escape (arXiv:2606.24506, full text), so some of the gap is plausibly the baseline’s known weakness rather than CrossPool’s intrinsic edge. Without head-to-head numbers against a Dynamo-class system in the same regime, the absolute magnitude is hard to place.
An early secondary write-up, from Machine Brief on 24 Jun 2026, ran the headline “CrossPool: The breakthrough for MoE Model Efficiency” and repeated the figure as a P99 TBT reduction “compared to traditional methods” without the model, request rate, testbed or baseline that would locate it. That is the fast-follow coverage to expect: abstract-restating and regime-blind. The arXiv primary is the source to anchor on.
What does the transfer path cost?
Disaggregating two memory pools means moving a tensor between them on every layer of every active request, and that movement is the tax prefill-decode serving already pays in production. The NVIDIA Dynamo documentation is explicit about where the bottleneck lands: without RDMA the system falls back to TCP, and KV-cache movement can dominate time-to-first-token and throughput. CrossPool’s hidden-state boundary inherits the same economics on a different tensor and a different cadence.
CrossPool’s countermeasure is its layer-wise pipeline scheduler, which overlaps the hidden-state transfer for one batch with compute on the batches already in flight, so the network and the compute units stay busy instead of one waiting on the other (arXiv:2606.24506, full text). That works to the extent the transfers can be hidden behind compute, which depends on available batch depth and interconnect bandwidth. On a fabric that cannot feed experts faster than the monolithic-pool contention it replaces, the disaggregation is a net cost rather than a win.
The prototype’s wire bounds the evidence, so it is worth naming. Hidden states move over NVLink using NVSHMEM, the two pools typically sit within one or two nodes, and the evaluation ran on five 40 GB A100s (arXiv:2606.24506, full text). The authors are candid about the residue: attention-to-FFN and FFN-to-attention transfers remain on the critical path even with the scheduler, and pipelining a short-running model against a long-running one can drag the faster model’s decode time down to the slower one’s. Their own mitigation is placement-level, grouping cold models with similar attention and FFN compute profiles before they share pools.
What changes for multi-tenant MoE serving economics?
CrossPool relocates the binding constraint rather than removing it. It moves the bottleneck from per-model VRAM residency to the interconnect that ships per-layer hidden states between the two pools, and that is a real change in where capacity planning has to land. A cluster that was GPU-memory-bound on cold checkpoints can become network-bound on the disaggregation path, and which of those is the cheaper problem to solve is an operator-specific question.
The contrast with other MoE-serving economics work sharpens what CrossPool is and is not. Placing MoE experts on AWS Lambda with Bayesian expert-popularity prediction cut the billed cost of all MoE layers by at least 75.67% versus CPU clusters (arXiv:2501.05313). That is a different lever: placement and billing model, not memory pooling. The two are complementary. One asks where an expert should run to minimize cost; the other asks how weight and KV memory should be organized so the active request can reach it.
What CrossPool does not do is make cold models free. It makes their co-residency cheaper by consolidating the memory that was reserved per model, and it pays for that with a transfer path that has to keep up. The durable insight, the part that outlives this paper’s specific numbers, is the lifetime mismatch: static weights and transient KV-cache live on different timescales, and provisioning for the worst case of each, per model, inside one pool is the structural mistake. CrossPool is one particular way to stop making it. Whether it is the right way for a given cluster comes down to the same wire the rest of disaggregated serving lives or dies on: how fast you can move the bytes.
Join the discussion
Share a useful perspective or ask a question about this article.