When prefill and decode disaggregation helps LLM serving

Prefill and decode disaggregation separates prompt processing from token generation. The split can reduce interference when a long prompt arrives while other requests are streaming, but it creates a handoff: the decode worker needs the prompt’s KV state before continuing generation. A deployment gains scheduling freedom and pays for transfer, routing, duplicated capacity, and another failure boundary. The DistServe paper explicitly couples phase placement to the cluster’s bandwidth and evaluates requests served within latency constraints. (DistServe paper)

When prefill and decode disaggregation helps LLM serving

Time to first token measures how long a request waits for output to begin. Inter-token latency measures the spacing of subsequent output tokens. Neither is equivalent to aggregate tokens per second. A service can produce many tokens while too many individual requests miss their response target.

DistServe studies phase interference and chooses parallelism and placement to improve the rate of requests served inside its latency constraints. Across OPT models from 13B to 175B on A100 GPUs, with chatbot, code-completion, and summarization workloads, it reports serving up to 7.4× more requests or a 12.6× tighter SLO than the baselines it tested. (DistServe paper) It kept more than 90% of requests within constraints while doing so. Those results are bounded to the paper’s models, workloads, hardware, and SLOs. They do not establish that every provider uses this architecture or that every co-located system fails both latency targets.

The interference itself is well documented, and adoption has spread. The DistServe authors’ retrospective reports that a single large prefill can inflate time per output token by 2–30× under bursty workloads even with chunked prefill as a mitigation, and counts NVIDIA Dynamo, llm-d, Ray Serve LLM, SGLang, vLLM, LMCache, and Mooncake among the production frameworks now shipping some form of disaggregation. (Hao AI Lab, UC San Diego. “Disaggregated Inference: 18 Months Later.” 2025) Adoption makes the question practical; it does not answer it for a specific workload.

Later work narrows when the split wins. The TaiChi paper (2025) compares aggregation and disaggregation directly and reports that aggregation performs best under tight time-to-first-token targets with relaxed per-token targets, disaggregation under strict per-token targets with relaxed first-token targets, and that under balanced targets neither approach delivers the best goodput without a hybrid of prefill-heavy and decode-heavy instances. (TaiChi paper) In practical terms: strict streaming targets favor separation, tight first-token targets with tolerant streaming favor co-location, and balanced targets are exactly the case the headline numbers cannot settle.

Current vLLM documentation makes the tradeoff particularly clear. Its disaggregated-prefilling feature runs prefill and decode in separate instances, uses a KV connector for the handoff, and is labeled experimental. It describes separate latency tuning and tail inter-token latency as motivations, while explicitly warning that the feature does not improve throughput. That warning should remain visible beside broader research results. (vLLM. “Disaggregated Prefilling (experimental).” vLLM Documentation)

Test the simpler scheduler first

Chunked prefill breaks a long prompt into smaller pieces and can schedule those pieces alongside decode work. vLLM’s current optimization guide says its V1 scheduler enables chunked prefill where possible and prioritizes decode. It describes a tuning tradeoff: smaller token batches can improve inter-token latency, while larger batches can improve time to first token. (vLLM optimization guide)

DistServe’s discussion section draws the boundary in its own words. In offline applications that are not latency-sensitive, it states that the effectiveness of DistServe “may be compromised,” and that “techniques such as chunked-prefill with piggyback [9, 3] may be preferred since it can fill each batch to the compute-bound threshold, thereby maintaining higher GPU utilization in every iteration.” And for “environments with only a few or even a single GPU,” it finds that “the design space for DistServe is significantly limited,” so “simpler architectural choices like non-disaggregated systems [32, 3] may reduce deployment complexity and optimize operational efficiency.” (DistServe paper)

This creates a useful first comparison. Measure a tuned co-located configuration on the intended traffic before assigning another fleet to prefill. If it already meets the latency and cost targets, phase separation needs another demonstrated benefit to justify its operational work. That is an inference from the documented alternatives, rather than a claim that one scheduler always wins.

A research middle path exists for small fleets. Nexus applies the disaggregation idea inside a single serving engine, dynamically reallocating one GPU’s compute between phases. Its motivating findings are that chunked prefill still interferes with decode, and that GPU resource allocation shows diminishing returns beyond a saturation point, which is what makes splitting one GPU between phases viable. (Nexus paper) It is a prototype reported in a paper, not a shipped engine feature; check what the deployed runtime actually supports.

Account for KV transfer with the right units

DistServe’s own arithmetic shows which units to watch. It reports that a single 512-token request on OPT-66B produces a KV cache of approximately 1.13GB, so an average arrival rate of 10 requests per second implies moving 11.3GB per second, or roughly 90Gbps, to keep the transfer invisible. (DistServe paper) Clusters with fast InfiniBand absorb that; where cross-node bandwidth is limited, DistServe instead places matching prefill and decode stages on the same node so transfers ride intra-node NVLink.

A worked example shows where such figures come from, using the same accounting as DistServe’s estimate. (DistServe paper) For an illustrative conventional attention cache, assume 80 layers, eight KV heads, head dimension 128, 4,096 cached tokens, and two bytes per stored value; these are hypothetical inputs, not a measured footprint for any particular model. The keys and values require:

2 × 80 × 8 × 128 × 4096 × 2 = 1,342,177,280 bytes

That is 1.25 GiB for one request, or 10 GiB for eight such requests, using the same accounting as DistServe’s estimate. (DistServe paper) Actual architecture, cache precision, sharding, block allocation, and reused state can change the transferred amount.

At an ideal 100 Gbit/s line rate, close to the 90Gbps DistServe computes for its own example, 1,342,177,280 bytes divided by 12.5 billion bytes per second gives about 107 milliseconds for one request’s assumed cache (DistServe paper). This is a transfer-time lower bound before protocol overhead, contention, scheduling, or setup, not a network benchmark. Mixing the eight-request total with the one-request size produces the wrong latency estimate.

Reported transfer costs span a wide range, which is the point. In DistServe’s evaluation, KV-cache transmission accounts for less than 0.1% of total latency even on OPT-175B, and more than 95% of requests see less than 30ms of delay, because the placement pins matching stages to one machine so transfers ride intra-node links. (DistServe paper) FlowKV, studying an NCCL-based transfer path, measures KV transfer at roughly a quarter of end-to-end latency for a 13,000-token input with 100 tokens of output, and redesigns its transfer method and allocator around that cost. (FlowKV Authors. “FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling.” arXiv, April 2025) Neither result proves that TCP is never viable or that one interconnect is mandatory for every production deployment. Placement and transfer method decide, and the papers make placement explicitly bandwidth-dependent.

Keep architecture gains tied to the whole experiment

Mooncake is a KV-centric architecture: its paper combines phase separation with storage, reuse, and scheduling decisions. Its workload and cache-locality assumptions belong with its results. A gain from that system cannot automatically be assigned to phase separation alone. (Mooncake paper)

Likewise, the SGLang team’s documented multi-node DeepSeek deployment combines prefill/decode separation with large-scale expert parallelism and other serving choices, reaching 52.3k input and 22.3k output tokens per second per node on 96 H100 GPUs. (LMSYS Org. “Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs.” May 2025) The same post reports that configuration’s limits: time to first token of 2–5 seconds and inter-token latency around 100ms, which the authors flag as needing further optimization for real-time use. (LMSYS Org) A throughput-optimized disaggregated cluster is not automatically a low-latency one.

NVIDIA’s Dynamo 1.0 release article describes an orchestration and production-serving stack, and reports serving up to 7× more requests on Blackwell in the SemiAnalysis InferenceX benchmark, for DeepSeek R1-0528 at FP4 with 1k/1k traffic and disaggregated serving combined with wide expert parallel on GB200 NVL72. (NVIDIA. “How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale.” March 2026) That is a vendor-published result from a third-party benchmark on a full-stack configuration. It does not establish that the disaggregated path is the best topology for every model, or that the gain isolates this one optimization. The release also extends the pattern to a third stage for multimodal workloads, separating image encode from prefill and decode. A 2025 framework paper pushes the same staging further, consolidating GPU-based vector search into a shared pool that runs alongside prefill and decode instances, on the finding that retrieval entangled with inference inflates tail latency. (Trinity paper)

Make the decision from a matched workload

Use the same model artifact, request trace, total serving resources, and acceptance criteria when comparing co-location, chunked prefill, and separate pools. Retain prompt and output length distributions, cache reuse, arrival rates, and concurrency. Then measure latency distributions and the fraction of requests meeting the target, not just a peak token rate.

Vendor data shows how much the answer moves with the regime. AMD’s MI300X guide with SGLang reports that a one-prefill, one-decode pair served up to 6.9× more goodput requests for a chatbot trace (3,200 tokens in, 800 out) under a 1,000ms first-token and 25ms per-token budget, but 2.23× for a decode-heavy trace (1,024 in, 2,048 out) under a 500ms first-token budget. (AMD ROCm. “Unleashing AMD Instinct MI300X GPUs for LLM Serving: Disaggregating Prefill & Decode with SGLang.”) Same architecture, different workload and SLOs, a threefold difference in reported benefit.

These are proposed evaluation criteria derived from the papers and runtime documentation. They are not results from an original benchmark.

Test a stalled producer, failed transfer, and unavailable decode worker as well. The router’s recovery behavior and the service’s response to incomplete state belong in the operating case. Splitting the phases is useful when the measured scheduling and latency benefits exceed those transfer and operational costs. It has no universal concurrency threshold, speedup, or hardware topology that can substitute for that evidence.