DeepSeek MoE Serving: Where Expert-Parallel All-to-All Time Goes

If your DeepSeek-class MoE deployment shows inter-token latency climbing as concurrency rises, the most likely culprit is not bandwidth and not compute. It is fixed synchronization cost inside the expert-parallel all-to-all, and the diagnostic to confirm it costs you an afternoon of measurement rather than a purchase order. A November 2026 NVIDIA preprint, NVLinkOneSided (arXiv:2611.13188), puts numbers on this: at a few tokens per GPU rank, the bytes involved would cross NVLink in under a microsecond, yet measured dispatch and combine round trips take tens of microseconds in every MoE layer. The gap is producer-consumer synchronization, not wire time. Everything quantitative from that paper is author-reported, unreplicated, and scoped to Blackwell GPUs inside a single NVLink domain, so treat its thresholds as hypotheses to re-measure on your stack, not constants. But the mechanism it documents is corroborated by independent serving research, and it changes how you should spend tuning time before you spend money.

The consequence worth internalizing up front: for self-hosted MoE models, the NVLink domain, not aggregate FLOPS, increasingly sets the serving cost floor. That shifts capacity planning toward interconnect topology, and it means the cheapest fix for a decode-latency problem is often re-placement and kernel choice, not more GPUs.

Split the symptom before blaming the all-to-all

Serving diagnostics conventionally separate two regimes: time to first token (TTFT), which characterizes prefill, and time per output token (TPOT), measured during decode (arXiv:2609.33385). This split matters because the all-to-all diagnosis applies asymmetrically. During prefill, each rank holds hundreds or thousands of tokens; the communication has plenty of data to move and plenty of compute to hide behind. During decode, each rank may hold only a handful of tokens per step, and there is little compute to overlap with.

So the first diagnostic move is simple: hold your deployment fixed, sweep concurrency, and watch TTFT and TPOT separately. The sweep’s job is to locate your per-rank token count on the batch-scaling curve, because that position decides which cost dominates. The preprint’s own table defines the fixed-cost signature: dispatch plus combine is roughly flat from one to 128 tokens per rank (18.1 microseconds of a 36.7 microsecond layer at one token per rank, versus 35.0 of 183.3 at 128), so a decode step where each rank holds few tokens carries a communication floor of tens of microseconds per MoE layer, and that floor is largest as a share of layer time exactly there. Independent work on MoE prefill confirms the other side of the curve: “For Prefill, as the batch size increases, the proportion of communication time increases” (SMoES, arXiv:2604.23996), meaning at prefill scales the communication share is bandwidth-flavored and grows with data volume, while at decode scales it is fixed-cost-flavored and grows with layer count and EP degree.

Anatomy of an EP decode step

Expert parallelism exists because the weights do not fit anywhere else. At FP8, sparse-MoE weight footprints run from 31 GB for Qwen3-30B to 742 GB for GLM-5; all but the smallest exceed a single H100’s 80 GB of HBM, which “forces Expert Parallelism (EP) across multiple GPUs and across multiple nodes for large models—paying a cross-node all-to-all for every decode step” (FORGE, arXiv:2610.16890). DeepSeek-V3, the reference point for this model class, routes each token to 8 of 256 experts (TopKV, arXiv:2607.28633).

Under EP, every MoE layer runs the same choreography twice: “every MoE layer requires two all-to-all communications, namely dispatch and combine” (NVLinkOneSided). Dispatch sends each token to the GPUs hosting its selected experts; combine gathers the expert outputs back. Multiply the per-layer round trip by the number of MoE layers and you get the per-token overhead that shows up in TPOT. The preprint defines a round trip as dispatch plus combine and measures 17.4 to 47.9 microseconds for it at one token per rank; a hypothetical stack of 60 MoE layers, each paying a round trip in that range, would spend roughly 1.0–2.9 milliseconds per generated token on communication alone, before any expert math. The layer count is an assumption and the arithmetic is illustrative, but the per-layer and per-step structure is measured.

The wrong mental model here is bandwidth. The preprint’s central observation is that “the bytes a rank sends would take less than a microsecond at the theoretical NVLink bandwidth, yet measured round trips of dispatch and combine take tens of microseconds (§5) in every MoE layer: the time goes to fixed costs that do not grow with the number of tokens, such as synchronization between producers and consumers on different GPUs.” Buying faster accelerators does nothing for a cost that is independent of both data volume and FLOPS.

There is also a second, quieter cost in how mainstream kernels implement the exchange. In two-sided designs such as NCCL and DeepEP (the communication library, not the expert-placement scheme; the literature unfortunately uses “DeepEP” for both), “the sender writes data into an intermediate buffer, from which the receiver copies or regroups it into the layout that the next kernel needs; this extra pass over the data adds latency at small batches and consumes memory bandwidth at large ones” (NVLinkOneSided). So even when you leave the fixed-cost regime, the two-sided path keeps charging you, in HBM bandwidth that the KV cache and expert GEMMs also want.

The batch-scaling curve is the diagnostic

Per-rank token count is the single most informative variable you can sweep. Three regimes emerge from the evidence:

The crossover between the first two rows is, per the preprint, sharp: DeepGEMM’s MegaMoE fuses dispatch, both grouped GEMMs, and combine into one persistent kernel where “NVLink transfers proceed during Tensor Core work, which separate kernels cannot do, so MegaMoE leads from 256 tokens per rank (§5.3)” (NVLinkOneSided). That 256-token figure is the authors’ own measurement on their stack; your model width, quantization, and EP degree will move it. The durable finding is the shape of the curve, not the exact crossing point.

The useful reframe from the same paper: overlap-based approaches reduce how much communication is exposed, while a cheaper all-to-all reduces the communication cost itself. In decode there is almost nothing to hide behind, which is why decode is where fixed costs dominate and where kernel-level communication improvements pay off most.

The knob map, in order of expected effect

Given the diagnosis, here is what I would touch, in the order the evidence supports.

EP degree versus the NVLink domain boundary. Ring-style all-to-all “time complexity… is proportional to the number of processes and a larger world size of expert parallel brings longer communication time” (Pipeline MoE, arXiv:2304.11414), and the overhead “scales with EP degree and persists even with high-bandwidth interconnects” (FORGE). Worse, an EP group that straddles nodes changes transport class entirely: in a DGX H100 cluster, bandwidth varies by over 72x across interconnect levels, and “using RDMA for a transfer that could use NVLink wastes 18× available bandwidth. Conversely, attempting NVLink for cross-node transfers fails entirely” (TopKV). The hard rule that survives every source is the boundary: keep EP inside one NVLink domain. Inside that domain, smallest-is-best does not always hold. On a GB200 NVL72 rack, the preprint measures Sentinel’s round trip staying at 16.6–17.8 microseconds as EP scales from 8 to 64 GPUs, and at 128 tokens per rank the MoE layer is faster at larger EP (103 instead of 182 microseconds on 64 GPUs) because each rank loads fewer expert weights. So shrink EP to fit the domain, then treat further shrinkage as a hypothesis to measure, not a default. Even within a node, placement is not free: multi-GPU interconnect studies found NUMA-like effects from NVLink topology and routing, such that “choosing the right GPU combination can impose considerable impact on GPU communication efficiency” (arXiv:1903.04611).

Expert placement. DeepSeek-V3 deployments already choose between WideEP (experts spread for load balance) and DeepEP-style replication for locality (TopKV). Topology-aware placement research pushes the same direction: steering expert replicas toward intra-node placement preserves load peak-shaving while cutting cross-node traffic, at under 1% online planning overhead in training scope (TAOT, arXiv:2608.03676), and a simple modality-aware expert arrangement converted cross-node communication into local computation for inference-time reductions of 4.0–8.3% across batch sizes 1–32 on a 16-GPU two-node Qwen3-MoE setup (SMoES). These are different stacks and scopes, but they agree on the direction: a token that never crosses a node boundary is the cheapest token.

The communication kernel. This is where the new preprint enters. NVLinkOneSided replaces the two-sided exchange with one-sided pushes: each source rank writes tokens directly into rank-major buffers on the target GPUs, no metadata exchange first, and grouped GEMMs read them in place. Its Sentinel protocol makes the data themselves the flags, avoiding the flag-word interleaving of NCCL’s LL protocol; a second protocol, Fence, targets larger batches. The authors report it faster than DeepEP at every batch size and ahead of MegaMoE at decoding batches. It ships in FlashInfer as flashinfer.moe_ep.NVLinkOneSidedAlltoAll, and the preprint reports integration into vLLM, SGLang and TensorRT-LLM; the flag names that expose EP degree, expert placement, or kernel selection differ per engine version, so confirm them against your engine’s documentation before treating any of this as a runbook. Those performance claims are NVIDIA-authored, Blackwell-scoped, and unreplicated as of this writing. The stated limitation matters as much as the speedup: “the EP group must fit in one NVLink domain: both protocols rely on direct loads and stores to peer memory and on 32-bit words that arrive whole, which RDMA does not offer,” and the dispatch workspace grows with EP size, trading HBM against the KV cache (NVLinkOneSided). Cross-node EP keeps paying a two-sided all-to-all every decode step no matter which of these kernels you pick.

When it is not the all-to-all

The diagnostic fails safely only if you know its boundaries. Three counter-regimes from the evidence:

• Offloading and small-system deployments. On hybrid CPU-GPU execution over PCIe, “PCIe transfer accounts for up to 78.1% of total execution time under hybrid execution” (DALI, arXiv:2602.03495). There the bottleneck is expert weight movement, and no all-to-all tuning touches it.
• Long contexts. DeepSeek-V3’s MLA compresses per-request KV cache to roughly 270 MB versus about 1.3 GB for a 70B dense model, yet aggregate KV transfer demand still exceeds 100 GB/s at production concurrency (TopKV). Long-context workloads dilute the MoE share of per-token time with attention and KV traffic; re-measure the mix before assuming the all-to-all still dominates.
• Speculative decoding. Verification steps can activate “the union of all per-node top-k expert sets, whose cardinality may substantially exceed k” (AcceptMoE, arXiv:2608.02989). Expert traffic per forward pass is therefore not comparable before and after enabling speculative decoding; any A/B latency attribution that crosses that boundary is measuring two different workloads.

There is also a more radical answer on the horizon of this literature: delete the all-to-all rather than optimize it. Pipeline MoE replaces the expert-parallel exchange with tensor slicing inside tensor parallelism, reporting a 76.5% improvement (about 1.8x) over the DPMoE baseline with no inter-node communication inside the MoE layer (arXiv:2304.11414). That result is training-scoped and unproven for latency-sensitive serving, but it is a reminder that EP itself is a choice, not a law.

What changes for capacity planning

If the fixed-cost diagnosis holds on your stack, the buying decision inverts. More GPUs of the same interconnect class do not reduce a per-step synchronization cost, and pushing the EP group across a node boundary raises it outright. What reduces it is confinement: an EP group that fits one NVLink domain, experts placed intra-node, and a dispatch/combine path matched to your per-rank token regime. The interconnect topology, not the accelerator’s FLOPS rating, sets the floor on per-token cost, and NVIDIA’s production NIXL library already selects NVLink versus RDMA backends per transfer (TopKV), which tells you the topology-aware view is becoming the default rather than a niche optimization.

My recommendation, with its conditions attached: before any hardware purchase, split TTFT from TPOT and sweep per-rank batch size on your own deployment. If decode TPOT shows tens of microseconds per MoE layer while each rank holds few tokens, you have the fixed-cost signature, and the cheap sequence is EP confinement to one NVLink domain, intra-node placement, then kernel selection, with fused kernels reserved for the hundreds-of-tokens regime where the preprint’s own data says they win. If the signature is absent, look at weight offload, KV traffic, and speculative decoding before touching the communication path at all. The one-sided results are promising and the mechanism is sound, but they are one team’s Blackwell numbers; the curve shape is the durable finding, and the crossover points are yours to measure.

Frequently Asked Questions

How can I determine if fixed synchronization costs are causing my latency issues?

So the first diagnostic move is simple: hold your deployment fixed, sweep concurrency, and watch TTFT and TPOT separately. The sweep’s job is to locate your per-rank token count on the batch-scaling curve, because that position decides which cost dominates.

What is the primary constraint for keeping expert parallelism efficient?

The hard rule that survives every source is the boundary: keep EP inside one NVLink domain. Inside that domain, smallest-is-best does not always hold.

What is the main limitation of the NVLinkOneSided protocol?

The stated limitation matters as much as the speedup: “the EP group must fit in one NVLink domain: both protocols rely on direct loads and stores to peer memory and on 32-bit words that arrive whole, which RDMA does not offer,” and the dispatch workspace grows with EP size, trading HBM against the KV cache