Disaggregated LLM Serving: Diagnosing KV Cache Transfer Collisions

When a disaggregated serving setup starts stalling, the expensive mistake is buying bandwidth before localizing the drop. Three different layers, the connector that moves KV blocks, the network fabric’s path assignment, and the prefill scheduler, can each produce the same visible symptom: time-to-first-token inflates and decode GPUs sit waiting. Each layer has a different fix, and two of the fixes make the others worse. This article maps those failure modes, using one new workshop paper for the fabric-layer mechanism, so that basis is flagged up front: the routing-collision diagnosis rests on a single author-reported paper, ARK, published at ACM SIGCOMM’s NAIC ‘26 workshop by Georgia Tech authors and evaluated only in packet-level simulation, with no production measurements and no independent replication. The triage procedure below stands on its own; the collision mechanism is the piece you should verify in your own fleet before acting on it.

Three layers, one symptom

A decode stall after prefill/decode disaggregation has at least three plausible origins, and they are easy to confuse because all three surface as TTFT growth:

1. Connector placement. If the KV cache is pulled from the prefill node after prefill finishes, the entire transfer is serialized before the first token. That is a scheduling property of the connector, not a network fault.
2. Fabric path collisions. Two large KV transfers hashed onto the same spine link congest each other even when aggregate bandwidth is ample. That is a path-assignment problem, not a capacity problem.
3. Scheduler saturation. Under bursty, heavy-tailed traffic, prefill nodes saturate while decode nodes idle. The stall is real, but the fabric is innocent; the Kairos preprint reports that “under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized” on a production-style 2P2D A100 cluster.

The practical consequence is that “KV transfer is slow” is not a diagnosis. It is three diagnoses wearing one coat, and the remedy for each, change transfer placement, control fabric paths, or rebalance scheduling, does nothing for the other two.

Why the fabric becomes the bottleneck after splitting

Disaggregation exists for cost reasons. The argument, laid out in the memory-heterogeneous serving preprint, is that HBM accounts for 18% of A100 manufacturing cost and 45% of B200 cost, while an A100 wastes over 97% of its HBM bandwidth during a 4K-token prefill, as the authors measure. Splitting prefill and decode lets each stage run on hardware matched to its arithmetic profile. The trade is explicit in the ARK paper: “The price of this architecture is a new dominant communication pattern: the key–value (KV) cache produced by a prefill GPU must be shipped over the data-center fabric to the decode GPU before token generation can begin.”

The volumes are large enough that this is not a rounding error. Per the optical KV cache network paper, Llama-3.1-70B in BF16 needs 320 KiB of KV cache per token, so a 64K-token context occupies roughly 21 GB. The same paper analyzed one P/D application in Alibaba’s production trace and found the hourly mean KV transfer time was a median 1.10 times the prefill time over one week. Transfer did not trail compute; it matched it. The memory-heterogeneous paper adds a second calibration point: Qwen3-32B at 8K context produces roughly 2 GB of KV state. Over a 100 Gbps RDMA fabric, it estimates a monolithic post-prefill transfer takes 100–200 ms, which the authors put at 10–15% of prefill latency, charged in full to TTFT under a naive schedule.

Two cautions on these numbers. They are author-reported measurements from preprints, not independent benchmarks. And they are specific to the stated models and link speeds; they calibrate orders of magnitude, not values for your topology.

Pull versus stream: the connector decides before the fabric sees a packet

Before any routing question arises, the transfer connector has already decided how visible the transfer will be in latency. The optical KV cache paper characterizes the two dominant patterns this way: “Mooncake streams the KV cache layer by layer during prefill, which hides the transfer while it ends within the prefill. The vLLM NIXL connector pulls it after prefill [21], which adds the whole transfer to TTFT, because decode emits the leading token.”

That distinction is the cheapest thing to check in a stall investigation. If your measured TTFT inflation is roughly equal to the full transfer time for the context lengths involved, the arithmetic from the previous section lets you estimate that, then the transfer is being serialized after prefill, and the fix is placement: stream layer by layer during prefill, or otherwise overlap transfer with compute. Bandwidth does not address the root of a serialization decision: a faster link shrinks the serialized transfer, but the bytes were never contending for the wire; they were queued behind a dependency, and only placement removes that wait.

The same paper quantifies the placement penalty on a contended fabric: at half of 400G per GPU, the median migrated turn of Llama-3.1-70B on H100s added 67 ms to TTFT on an oversubscribed packet fabric. Note that this figure bundles placement and contention together, which is exactly the confound a triage procedure needs to separate.

One provenance caveat applies to everything in this section: these characterizations of vLLM’s NIXL connector and Mooncake come from a third-party paper, not first-party documentation. Connector semantics can differ by release, so treat this as a hypothesis to confirm against the version you actually run.

What ARK actually claims about routing collisions

The newest piece of the failure map is the fabric layer itself. ARK’s paper argues that “AI Clos fabrics commonly oversubscribe the leaf–spine layer for cost reasons, which further amplifies collision impact: with less spare bandwidth, collision-induced queueing fills buffers faster.” The mechanism is ECMP hash collision: when two large KV transfers hash onto the same equal-cost path, they share one spine link while other paths sit idle. Because KV transfers are large, synchronized, and latency-critical, the collisions land directly on TTFT and decode start.

ARK’s sharper claim is about what existing optimizations do not address: “None of these approaches controls which equal-cost path the remaining bytes traverse once admitted to the fabric—a dimension that ARK addresses and that compounds with, rather than competes against, the bytes-on-wire reductions above.” The bytes-on-wire reductions it names, MemServe, LMCache, CacheGen, DualPath, shrink how much KV crosses the fabric. ARK’s position is that path assignment is an orthogonal dimension, so cache compression and path control stack rather than substitute.

This is the load-bearing caveat of the article, and it belongs in plain sight. ARK’s evidence is packet-level simulation and nothing more. The paper reports that “Packet-level RDMA simulations show that ARK reduces mean and P95 FCT by up to 27.3% and 34.0% under moderate load, and further reduces mean TTFT by up to 12.9%,” measured against vanilla ECMP on a 16-GPU two-tier fat-tree in SimAI/NS-3 with a Llama-3-8B Zipf workload. The paper reports no production deployment, and the result has no independent replication. What the paper establishes is a plausible, mechanistically specific failure mode consistent with known Clos oversubscription economics, plus simulated evidence that fixing it pays when the fabric is congested but not saturated. Whether collisions actually degrade your fleet’s TTFT is an empirical question the paper cannot answer for you. The implication for spending is direct: if collisions are your problem, added bandwidth is not the fix, because the problem is which path bytes take, not how much aggregate capacity exists. But “if” is doing real work in that sentence.

Sidestepping the fabric instead of fixing it

Two recent preprints argue the routing problem is avoidable by removing the transfer from the critical path, or removing the network hop entirely. Both deserve weight because they directly contest the premise that fabric path control is where to invest.

Request deflection. Kairos routes requests so the KV cache is built where it will be consumed: “Kairos’s deflection path eliminates the KV transfer cost because the KV cache is built directly on the decode node.” The authors report P95 TTFT reductions of up to 81% and SLO attainment gains of up to 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost. Those figures are author-reported and from a single preprint. More important is the trade the paper concedes: “However, if a request has a prefix already cached on a prefill node, deflecting it to a decode node loses that cached prefix. Kairos currently does not account for prefix cache residency in its TTFT estimate.” Deflection is free only when nothing reusable is left behind. In a workload with heavy prefix sharing, multi-turn conversations against a common system prompt are the canonical case, discarding the cached prefix means re-prefilling, which can cost more than the transfer ever did.

Rack-scale CXL shared memory. TraCT replaces the fabric rather than optimizing its paths, using rack-scale CXL shared memory as both the KV transfer substrate and a rack-wide prefix-aware cache, with GPUs reading and writing KV blocks via CXL load/store and DMA. The authors conclude that “CXL shared memory is a viable transport substrate for rack-scale disaggregated LLM inference and eliminates the network hop that dominates existing prefill–decode pipelines.” Where ARK accepts the NIC-hop fabric and fixes its path assignment, TraCT removes the hop, which removes ECMP collisions by construction but binds you to rack-scale scope and a memory-substrate roadmap rather than a networking one.

The conflict between these papers and ARK is workload-dependent, not settled. In a 2P2D cluster under bursty load, Kairos found the scheduler was the binding constraint. ARK’s collision argument matters most when many large transfers run concurrently across an oversubscribed Clos, a regime where disaggregation is already deployed widely: the ARK paper notes Mooncake serves Moonshot AI’s Kimi chatbot with disaggregated pipelines in production, and SGLang’s reference deployment of DeepSeek-V3 on 96 H100 GPUs adopts prefill, decode disaggregation.

Matching the remedy to the layer

The evidence supports a symptom-driven triage, summarized here:

Two spending traps follow from the table. First, more transfer bandwidth does not address the root cause in any of the top three rows: row one is serialization (a faster link shrinks the wait but leaves it serialized), row two is path assignment, row three is scheduling. Second, tighter colocation addresses the fabric rows but gives up the hardware-cost asymmetry that motivated splitting in the first place.

What remains unverified, and how to check your own fleet

The honest state of the evidence: the connector placement distinction and the production transfer costs come from author-reported preprints; the collision mechanism and its gains come from one workshop paper evaluated only in simulation, with no production measurements and no public replication; and every description of vLLM, Mooncake, and NIXL behavior here comes from third-party papers, not first-party vendor documentation, so confirm connector semantics against the release you run before treating any of this as configuration guidance.

If I were triaging a stall with this map, I would start with the cheapest discriminating measurement: compare observed TTFT inflation against the expected full transfer time (context length times per-token KV bytes, divided by effective link throughput). A match points at placement. If placement is already streamed, the next question is whether stalls correlate with concurrent transfer count rather than total offered load; that pattern is the fabric-collision signature ARK predicts, and it is testable with flow-level telemetry before spending anything on path control. Only after both of those come back negative would I look at the scheduler, where the signature is prefill saturation alongside idle decode compute. The order matters because each test is cheaper than the fix it gates, and because the one mechanism here backed by a single simulation-only paper is also the one most likely to justify an expensive fabric project.

Frequently Asked Questions

How can I tell if the stall is caused by connector placement rather than the network?

If your measured TTFT inflation is roughly equal to the full transfer time for the context lengths involved, the arithmetic from the previous section lets you estimate that, then the transfer is being serialized after prefill, and the fix is placement: stream layer by layer during prefill, or otherwise overlap transfer with compute.

What is the main limitation of the ARK paper’s findings on routing collisions?

ARK’s evidence is packet-level simulation and nothing more. The paper reports that “Packet-level RDMA simulations show that ARK reduces mean and P95 FCT by up to 27.3% and 34.0% under moderate load, and further reduces mean TTFT by up to 12.9%,” measured against vanilla ECMP on a 16-GPU two-tier fat-tree in SimAI/NS-3 with a Llama-3-8B Zipf workload. The paper reports no production deployment, and the result has no independent replication.

What is a downside of using request deflection to avoid KV transfer costs?

However, if a request has a prefix already cached on a prefill node, deflecting it to a decode node loses that cached prefix. Kairos currently does not account for prefix cache residency in its TTFT estimate.