What expert offloading actually costs is a trade rather than a number: you exchange VRAM capacity for host-memory bandwidth, and the bill arrives every token. That trade is the design argument of FluxMoE, a preprint that streams mixture-of-experts weights from host DRAM on demand while keeping expert computation on the GPU. Its current abstract (v3, posted 2026-09-10) does report numbers: up to 7.2x vLLM’s throughput and 79.0% lower average Time-Per-Output-Token (TPOT) for GLM-4.5 on 8x H20 GPUs (abstract), and 4.3x the throughput of the KTransformers baseline for Mixtral-8x7B-Instruct on 2x L40S GPUs. Those are author-reported figures, no independent replication appears in this evidence set, and both reported setups are multi-GPU, which matters for how far the results travel.
The distinction worth holding onto is author-reported measurement versus independently checked result. FluxMoE’s authors say they implemented it atop vLLM and evaluated it on three MoE models; the abstract names two of them with their hardware, GLM-4.5 on 8x H20 and Mixtral-8x7B-Instruct on 2x L40S, and leaves the third unnamed (arXiv:2604.02715). The baseline stack underneath is real and documented: vLLM’s repository lists mixture-of-experts support for Mixtral, DeepSeek-V3, Qwen-MoE, and GPT-OSS. And the ecosystem now has commercial stakes, since vLLM’s creators reportedly raised a $150M seed round for their startup Inferact in January 2026 (Wikipedia’s vLLM entry, citing TechCrunch). None of this confirms FluxMoE’s numbers outside the authors’ own runs. It makes them worth pricing out carefully.
Why 17B of compute can demand 400B of storage
An MoE model separates two budgets that a dense model keeps joined. Memory scales with total parameters, because every expert’s weights must be stored somewhere. Compute scales with the active parameters, the subset consulted per token.
Llama 4 Scout sits at the accessible end of that split: 109B parameters stored, 17B active per token, roughly 6.4x more weight to hold than compute to run, according to Ollama’s Llama 4 page. Maverick stores 400B against the same 17B active per token, about 23.5x by the same arithmetic (Ollama).
| Model | Total parameters | Active per token | Storage-to-compute ratio |
|---|---|---|---|
| Llama 4 Scout | 109B | 17B | ~6.4x |
| Llama 4 Maverick | 400B | 17B | ~23.5x |
DeepSeek-V3, Qwen-MoE, Mixtral, and GPT-OSS are the other MoE targets vLLM documents; the retrieved sources give no parameter splits for them, so the memory math here stops at Llama 4.
The single-GPU problem follows directly from the table. A card with comfortable headroom for 17B parameters of per-token work can still be unable to hold the model, because capacity is charged against the total. What creates room to maneuver is the premise the abstract itself states, “sparse expert activation”: only a subset of a model’s experts fires per token. A residency policy is a bet that the consulted subset stays small enough to plan around, and FluxMoE’s budget-aware residency planner exists to make that bet. Whether a given workload concentrates its expert selections that way is an empirical question; the abstract reports throughput and TPOT, not selection-frequency distributions.
Three offloading strategies, three different bills
FluxMoE’s abstract names its alternatives, CPU-GPU co-inference and whole-layer offloading (arXiv:2604.02715). The useful way to compare all three approaches is to ask what crosses the CPU-GPU boundary, and in which direction.
| Strategy | Where expert math runs | What crosses the bus | Governing cost | Status |
|---|---|---|---|---|
| CPU-GPU co-inference | Split between CPU and GPU | Activations and partial results | CPU compute throughput | Named as an alternative in the abstract; no figures retrieved |
| Whole-layer offloading | CPU, for offloaded layers | Activations at layer boundaries | CPU compute, coarse granularity | Named as an alternative; no figures retrieved |
| Expert paging (FluxMoE) | GPU | Expert weights, fetched on demand | Host DRAM bandwidth and interconnect | Author-reported: up to 7.2x vLLM’s throughput and 4.3x KTransformers’s throughput; no independent replication |
Co-inference moves computation to the weights: selected experts execute on the CPU, and the price is CPU throughput on some of the heaviest matrix multiplications in the model. Whole-layer offloading does the same at coarser granularity, moving entire layers to the host. Coarse units are simpler to manage but cannot exploit expert-level skew; a layer’s frequently and rarely used experts travel together. Expert paging inverts the direction and moves weights to the computation: expert math stays on the GPU while non-resident expert weights stream across, with the price paid in bytes per second rather than FLOPs per second.
Here the evidence has shape but no second opinion, and it should be said plainly. The abstract’s figures are the authors’ own: up to 7.2x vLLM’s throughput and 79.0% lower average TPOT for GLM-4.5 on 8x H20 (arXiv:2604.02715), and 4.3x KTransformers’s throughput and 29.1% lower TPOT for Mixtral-8x7B-Instruct on 2x L40S (arXiv:2604.02715). One calibration gap deserves naming: the second baseline in the paper is KTransformers, not llama.cpp. No llama.cpp CPU-offload measurement exists in the retrieved evidence, so the 4.3x figure cannot be mapped onto that comparison. Nor are there figures for the co-inference and whole-layer alternatives, or any third-party run of FluxMoE. Structural reasoning identifies what each strategy pays; only the authors’ own measurements rank any of the totals.
Inside expert paging: what moves and what stays put
The design’s stated target is the memory status quo: per the abstract, existing inference engines keep all experts GPU-resident, which crowds out the key-value cache in large-batch, long-output offline workloads (arXiv:2604.02715). Against that, the preprint describes three components. PagedTensor provides transparent remapping: the inference path addresses experts through stable logical slots while their physical residency moves between compressed VRAM and host DRAM underneath, the way a process addresses virtual memory while the operating system decides what is resident. The storage hierarchy is bandwidth-balanced across two tiers, losslessly compressed GPU memory and host DRAM. Lossless matters here because it shrinks the footprint without changing a single output bit, unlike lossy size reductions, and the authors report no measurable model-quality loss from it. A budget-aware residency planner then decides which experts occupy the expensive tier given a memory budget.
The economic shape of the design is what stands out. Weights are not like requests: they are re-read every time they are used, so a streamed expert’s transfer cost recurs with its selection frequency rather than amortizing away. That recurrence is what relocates the bottleneck. Capacity asks whether the model fits; bandwidth asks how fast you can refill the working set. FluxMoE’s wager is that refill, not fit, is the cheaper question when the alternative is not serving the model at all. The Mixtral result points at exactly that case: on 2x L40S, per the abstract, weight-resident vLLM cannot fit the model, and FluxMoE still reports 4.3x KTransformers’s throughput.
One mechanism the abstract leaves unexamined is concurrency. A batch of N requests can, in principle, touch up to N times as many distinct experts per layer per step, multiplying streaming traffic exactly when throughput matters most. Whether the residency planner absorbs that pressure is a question the abstract’s summary figures do not break out; they report aggregate throughput and TPOT, not behavior across batch sizes.
What the preprint establishes, and what it cannot
Established at the design level and reported by the authors (arXiv:2604.02715):
- The expert paging abstraction, which decouples expert residency from the inference path and adapts the expert footprint to available memory.
- An implementation atop vLLM, the serving stack practitioners already run.
- An evaluation on three MoE models, two named in the abstract: GLM-4.5 on 8x H20 and Mixtral-8x7B-Instruct on 2x L40S.
- Author-reported throughput and TPOT gains against two baselines: the 7.2x-vLLM and 4.3x-KTransformers results quoted above, without measurable model-quality loss using lossless compression.
Not established:
- The identity of the third evaluated model.
- Any VRAM-footprint figure; the cached abstract reports throughput and TPOT only.
- Any independent replication of the reported figures.
- Any measured comparison against llama.cpp CPU offloading, or any figures at all for CPU-GPU co-inference and whole-layer offloading.
The baseline stack context is solid ground: vLLM originated in UC Berkeley’s Sky Computing Lab and counts over 2,000 contributors (vLLM on GitHub), which makes it the natural comparison point for MoE serving work. The same centrality cuts the other way. Results produced inside the vLLM orbit deserve the same skepticism applied to vendor benchmarks, particularly with Inferact’s reported $150M seed round commercializing the project (Wikipedia). A preprint whose only measurements are its authors’ own, built on a commercially contested stack, deserves that skepticism twice over.
Is one GPU plus host DRAM enough?
The paper cannot answer that directly, because both of its reported setups are multi-GPU: 8x H20 for GLM-4.5 and 2x L40S for Mixtral-8x7B-Instruct. No single-GPU configuration appears in the abstract. Treat what follows as a planning extrapolation from the design, useful for one card or a small fleet, not as a measured result. The architecture tells you which questions to ask, in order:
- Does the total expert weight set fit in VRAM after lossless compression? If yes, residency policy is moot; run everything resident.
- If not, what does your workload’s expert-selection pattern look like? Measure selection frequency on representative traces. If the working set fits the compressed VRAM budget, paging-style streaming becomes plausible, and your planning currency shifts to host DRAM capacity and interconnect bandwidth, on the premise that host DRAM is the larger pool on a single-GPU machine.
- If the working set itself does not fit, expect to live with co-inference or whole-layer offload economics, or move to a smaller model. The bandwidth bill would exceed the capacity relief.
These are planning heuristics derived from the design, not measured thresholds. No number in the abstract separates the rungs, and the step that matters most, step 2, is a property of your traffic rather than of the system.
The verdict, and the evidence that would move it
For a single GPU or a small fleet serving DeepSeek- or Qwen-class MoE models, the actionable conclusion from this evidence is to plan around an expert-residency policy rather than raw VRAM capacity: a working set of experts held in losslessly compressed VRAM, the remaining expert weights streamed from host DRAM on demand, expert computation kept on the GPU. Accept that host-memory bandwidth becomes the governing cost, and budget for it the way you currently budget VRAM. That conclusion extrapolates from the paper’s design; the paper’s own measurements come from multi-GPU machines.
Treat FluxMoE’s numbers as a strong author-reported signal on two baselines, not deployment guidance. Its limitations are specific: the third evaluated model is unnamed in the abstract, no VRAM-footprint figure appears there, the llama.cpp CPU-offload measurements that would calibrate the offloading comparison are missing, no independent replication exists (arXiv:2604.02715), and every reported configuration is multi-GPU. Nothing in it generalizes to a specific single-GPU model and card combination.
What would move the verdict: independent replication naming the third model, with throughput, latency, and VRAM-footprint figures; llama.cpp CPU-offload measurements on the same machines; and third-party runs, since vLLM-ecosystem results now carry commercial weight. Until any of that exists, the cost of expert offloading is something you can reason about structurally and check against two author-reported benchmarks, but not against an independent one.
Frequently Asked Questions
What are the author-reported performance gains for FluxMoE compared to vLLM and KTransformers?
Its current abstract (v3, posted 2026-09-10) does report numbers: up to 7.2x vLLM’s throughput and 79.0% lower average Time-Per-Output-Token (TPOT) for GLM-4.5 on 8x H20 GPUs (abstract), and 4.3x the throughput of the KTransformers baseline for Mixtral-8x7B-Instruct on 2x L40S GPUs.
Does the FluxMoE preprint include results for a single-GPU configuration?
The paper cannot answer that directly, because both of its reported setups are multi-GPU: 8x H20 for GLM-4.5 and 2x L40S for Mixtral-8x7B-Instruct. No single-GPU configuration appears in the abstract.
What is the difference between Llama 4 Scout and Maverick in terms of storage-to-compute ratio?
Llama 4 Scout sits at the accessible end of that split: 109B parameters stored, 17B active per token, roughly 6.4x more weight to hold than compute to run, according to Ollama’s Llama 4 page. Maverick stores 400B against the same 17B active per token, about 23.5x by the same arithmetic (Ollama).
