When prefill and decode disaggregation helps LLM serving

Prefill and decode disaggregation lets LLM serving separate prompt processing from token generation. Its value depends on latency targets, KV transfer, and worker utilization.

The separation can reduce interference when a long prompt arrives while other requests are streaming. It also creates a handoff: the decode worker needs the prompt’s KV state before continuing generation. A deployment gains scheduling freedom and pays for transfer, routing, duplicated capacity, and another failure boundary. The DistServe paper explicitly couples phase placement to the cluster’s bandwidth and evaluates requests served within latency constraints. (DistServe paper)

When prefill and decode disaggregation helps LLM serving

Time to first token measures how long a request waits for output to begin. Inter-token latency measures the spacing of subsequent output tokens. Neither is equivalent to aggregate tokens per second. A service can produce many tokens while too many individual requests miss their response target.

DistServe studies phase interference and chooses parallelism and placement to improve the rate of requests served inside its latency constraints. Its results are bounded to the paper’s models, workloads, hardware, and SLOs. They do not establish that every provider uses this architecture or that every co-located system fails both latency targets. (DistServe paper)

Current vLLM documentation makes the tradeoff particularly clear. Its disaggregated-prefilling feature runs prefill and decode in separate instances, uses a KV connector for the handoff, and is labeled experimental. It describes separate latency tuning and tail inter-token latency as motivations, while explicitly warning that the feature does not improve throughput. That warning should remain visible beside broader research results. (vLLM. “Disaggregated Prefilling (experimental).” vLLM Documentation)

Test the simpler scheduler first

Chunked prefill breaks a long prompt into smaller pieces and can schedule those pieces alongside decode work. vLLM’s current optimization guide says its V1 scheduler enables chunked prefill where possible and prioritizes decode. It describes a tuning tradeoff: smaller token batches can improve inter-token latency, while larger batches can improve time to first token. (vLLM optimization guide)

This creates a useful first comparison. Measure a tuned co-located configuration on the intended traffic before assigning another fleet to prefill. If it already meets the latency and cost targets, phase separation needs another demonstrated benefit to justify its operational work. That is an inference from the documented alternatives, rather than a claim that one scheduler always wins.

The related KV transfer connector discussion covers a separate implementation detail. Use the current runtime’s connector and compatibility documentation for the exact version being deployed.

Account for KV transfer with the right units

For an illustrative conventional attention cache, assume 80 layers, eight KV heads, head dimension 128, 4,096 cached tokens, and two bytes per stored value. The keys and values require:

2 × 80 × 8 × 128 × 4096 × 2 = 1,342,177,280 bytes

That is 1.25 GiB for one request, or 10 GiB for eight such requests. These dimensions are stated hypothetical inputs, not a measured cache footprint for every 70B model. Actual architecture, cache precision, sharding, block allocation, and reused state can change the transferred amount.

At an ideal 100 Gbit/s line rate, 1,342,177,280 bytes divided by 12.5 billion bytes per second gives about 107 milliseconds for one request’s assumed cache. This is a transfer-time lower bound before protocol overhead, contention, scheduling, or setup, not a network benchmark. Mixing the eight-request total with the one-request size produces the wrong latency estimate.

Whether that cost is acceptable depends on the latency budget and the actual transfer method. RDMA and other fast paths can help, but the calculation does not prove that TCP is never viable or that one interconnect is mandatory for every production deployment. DistServe makes placement bandwidth-dependent; FlowKV studies transfer and scheduling mechanisms rather than a universal link requirement. (DistServe paper; FlowKV Authors. “FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling.” arXiv, April 2025)

For another model-specific handoff question, see KV transfer cost. It should be evaluated with that model’s actual state and weight-placement design.

Keep architecture gains tied to the whole experiment

Mooncake is a KV-centric architecture: its paper combines phase separation with storage, reuse, and scheduling decisions. Its workload and cache-locality assumptions belong with its results. A gain from that system cannot automatically be assigned to phase separation alone. (Mooncake paper)

Likewise, the SGLang team’s documented multi-node DeepSeek deployment combines prefill/decode separation with large-scale expert parallelism and other serving choices. It is a concrete deployment study, rather than evidence that every listed organization or runtime has adopted the same setup. (LMSYS Org. “Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs.” May 2025)

NVIDIA’s Dynamo 1.0 release article describes an orchestration and production-serving stack. A release-level production claim for that stack does not establish that its disaggregated path is the best topology for every model or that a quoted full-stack gain isolates this one optimization. (NVIDIA. “How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale.” March 2026)

Make the decision from a matched workload

Use the same model artifact, request trace, total serving resources, and acceptance criteria when comparing co-location, chunked prefill, and separate pools. Retain prompt and output length distributions, cache reuse, arrival rates, and concurrency. Then measure latency distributions and the fraction of requests meeting the target, not just a peak token rate.

These are proposed evaluation criteria derived from the papers and runtime documentation. They are not results from an original benchmark.

Test a stalled producer, failed transfer, and unavailable decode worker as well. The router’s recovery behavior and the service’s response to incomplete state belong in the operating case. Splitting the phases is useful when the measured scheduling and latency benefits exceed those transfer and operational costs. It has no universal concurrency threshold, speedup, or hardware topology that can substitute for that evidence.