Request Routing for Disaggregated LLM Serving: Where Learned Routers Win

A new arXiv preprint argues that learned request routing for disaggregated LLM serving only earns its keep after hardware calibration. The author-reported results, from a single study with no independent replication, show a calibrated router reaching 0.864 mean goodput against 0.835 to 0.847 for three heuristic baselines, and matching round-robin goodput with six GPUs instead of seven. Below roughly three decode instances, the advantage disappears.

What the study actually measured

The paper, “Calibrate, Then Route” (arXiv:2609.16206, feed-observed 2026-09-17), tackles a problem that only exists once you split serving in two. Disaggregated serving puts prompt processing (prefill) and token generation (decode) on separate GPU pools, moving KV caches between them over the network. vLLM already lists “disaggregated prefill, decode, and encode” among its serving features, per the project’s GitHub repository, so this is a shipped configuration, not a lab construct. Once prefill and decode are separate pools, every incoming request needs a routing decision: which decode instance gets it, given each instance’s queue, cache state, and expected cost.

The authors developed their routing policy in a discrete event simulator and then validated it on real hardware: eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools, according to the preprint. NIXL is the same transfer library vLLM adopted for its split deployments; Groundy covered the v0.21 bi-directional KV transfer work earlier this year. The evaluation ran three mixed, bursty arrival traces against three baseline heuristics: round robin, least loaded, and a length heuristic.

Two scope notes matter before any numbers. The validation used vLLM engines only. Teams searching for SGLang or llm-d routing guidance will find none in this evidence; whether the calibration argument transfers to those stacks is untested here. And the hardware was a single eight-A40 configuration, which constrains how far the specific constants generalize.

The goodput scoreboard

The headline result is real but modest. Across the three traces, the calibrated learned router achieved the highest mean goodput at 0.864, versus a range of 0.835 to 0.847 for round robin, least loaded, and the length heuristic, the authors report. That is a margin of roughly 1.7 to 2.9 goodput points over the baselines, not an order-of-magnitude shift.

The per-trace breakdown is the honest version of the story. The calibrated router beat round robin and the length heuristic on all three traces and least loaded on two; on the third it trailed least loaded by 0.003, which the authors themselves describe as within run-to-run noise, per the preprint. So the defensible claim is not “learned routing beats heuristics.” It is “a calibrated learned router is consistently at or near the top, with its clearest gains under mixed, bursty traffic.”

One thing this evidence does not settle: prefix-affinity routing. The fetched results test round robin, least loaded, and a length heuristic. The preprint may address cache-affinity-aware baselines in its full text, but the evidence available here does not, so claims that learned routing displaces prefix-affinity heuristics specifically should be treated as open until verified against the complete paper.

Calibration is the real bottleneck

The paper’s sharpest finding is not that learned routing wins. It is that an uncalibrated learned router is worse than useless effort. When the authors substituted simulator-derived constants for hardware-measured ones, the router lost 4.5 goodput points and roughly 40 percent of its tail latency advantage, “reducing the scorer to little more than queue counting,” in the paper’s own words.

The mechanism is straightforward. A learned router scores candidate decode instances by predicted request cost, and that prediction depends on constants: how long prefill of a given length takes, how KV transfer time scales, how decode speed degrades with batch size. A simulator can produce plausible constants, but plausible is not measured. On the actual A40-plus-NIXL setup, the simulator’s numbers were wrong enough that the scorer’s rankings carried almost no information beyond which instance had the shortest queue. You pay the complexity of a learned system and get back a queue counter.

The consequence for operators is a reallocation of effort. The bottleneck in routing quality is not GPU capacity or scheduler sophistication; it is calibration data collected on your hardware, under your traffic. And because the paper’s own argument implies these constants are hardware-specific, a router calibrated on A40s should be assumed uncalibrated on anything else, whether that is H100s or Grace Blackwell systems, until re-measured. There is no evidence in this study for any cross-hardware transfer of the calibrated constants.

Where learned routers stop winning

The authors are unusually direct about their result’s boundary: benefits “grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough,” the preprint states.

Both halves of that sentence deserve weight. Small pools first: with three decode instances, the routing decision has so few options that queue length nearly determines the right answer. A learned scorer adds model overhead, calibration burden, and a new failure mode, and returns nothing measurable. If your split deployment is a prefill node feeding two or three decode engines, this paper is an argument for leaving the scheduler default alone.

Traffic homogeneity is the quieter constraint. The evaluation traces were deliberately mixed and bursty, the conditions where cost prediction matters most because requests differ widely in prefill length and decode duration. If your workload is uniform, say a single application issuing similar prompts, the variance a learned router exploits shrinks, and the paper gives no evidence it would win there. The measured 0.864 figure is best read as a hypothesis about favorable conditions: heterogeneous, bursty arrivals against a decode pool large enough for routing choices to matter.

The capacity-planning math

The most quotable result for infrastructure teams is the GPU equivalence: with calibrated costs, the learned router matched the goodput of round robin using six GPUs instead of seven, according to the authors. Framed as a capacity claim, better routing bought roughly one GPU’s worth of headroom out of seven, about 14 percent, on this configuration.

Treat that number as directional, not as a procurement rule. It comes from the same single hardware setup and the same three traces as everything else in the study, and goodput equivalence at one operating point does not guarantee the same ratio at different load levels or pool geometries. What it usefully establishes is the shape of the trade: router quality is a measurable, tunable input to capacity planning, sitting alongside GPU count rather than fixed inside the scheduler. A team deciding between an eighth GPU and a calibration effort now has author-reported evidence that the calibration path can pay, at least once the pool is large enough for routing to matter.

That reframing also lands in a specific commercial moment. vLLM, originally developed in the Sky Computing Lab at UC Berkeley and now backed by over 2,000 contributors per the project repository, became a commercial venture when TechCrunch reported in January 2026 that its creators had launched Inferact with $150 million in seed funding, per Wikipedia’s vLLM entry. Serving efficiency is now a funded product category, and routing policy is one of the few remaining levers that does not require buying more hardware. Teams already running split deployments can measure their own baseline before touching anything; Groundy’s earlier coverage of prefill-decode disaggregation economics and vLLM cold-start behavior covers adjacent pieces of the same capacity puzzle.

How much to trust one preprint

All of the numbers above carry the same provenance, and it deserves plain statement. arXiv moderates submissions for topicality and scholarly value but does not peer-review them; contents are wholly the submitter’s responsibility, presented “as is,” per arXiv’s own about page. This study is author-reported, validated on exactly one hardware configuration, across three traces, with no independent replication as of 2026-09-17.

The internal consistency of the paper cuts both ways. The authors report their own boundary conditions, including the trace where least loaded matched them within noise and the pool sizes where their router adds nothing, which is the pattern of an honest measurement study rather than a marketing benchmark. But honest self-reporting is still self-reporting. The 0.864 goodput figure, the 4.5-point calibration penalty, and the six-versus-seven GPU equivalence should all be treated as starting hypotheses for your own measurement, not as specifications. Generalization to other engines, other hardware, and production traffic patterns is unproven.

Decision guide: when a calibrated router is worth the work

The evidence supports a fairly crisp decision sequence for teams running or planning disaggregated vLLM deployments:

1. Count your decode instances. At roughly three or fewer, the paper’s own results say queue-based baselines are competitive. Keep least loaded, spend the effort elsewhere.
2. Profile your traffic. The measured gains appeared under mixed, bursty, heterogeneous arrivals. Uniform workloads have less variance for a learned scorer to exploit, and this study offers no evidence of benefit there.
3. Budget calibration as a line item, not an afterthought. The 4.5-goodput-point penalty for simulator-derived constants is the paper’s strongest single result. If you cannot measure cost constants on your actual hardware, a learned router degrades to queue counting with extra moving parts, per the preprint.
4. Recalibrate when hardware changes. The calibration argument implies constants are hardware-specific. A router tuned on one GPU generation or interconnect should be assumed stale on another until re-measured.
5. Measure against least loaded, not just round robin. Least loaded was the strongest baseline and matched the learned router within run-to-run noise on one of three traces. That is the bar your own evaluation must clear.

The practical verdict: calibrate first, then route, and only if your decode pool justifies it. On this evidence, hardware-calibrated learned routing earns roughly 1.7 to 2.9 goodput points over round robin, least loaded, and a length heuristic, and can match round-robin goodput with six GPUs instead of seven. Without calibration, it collapses toward queue counting; below about three decode instances, queue counting was already enough. Routing quality is now a measurable component of capacity planning, but the measurement obligation lands on you, because the only numbers on record come from one team, one preprint, and eight A40s.

Frequently Asked Questions

How many GPUs did the calibrated router save compared to round robin?

with calibrated costs, the learned router matched the goodput of round robin using six GPUs instead of seven, according to the authors.

What happens to a learned router if it uses simulator-derived constants instead of hardware measurements?

When the authors substituted simulator-derived constants for hardware-measured ones, the router lost 4.5 goodput points and roughly 40 percent of its tail latency advantage, “reducing the scorer to little more than queue counting,” in the paper’s own words.

At what pool size does the advantage of learned routing disappear?

benefits “grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough,” the preprint states.