Expert streaming can make a 180B-parameter mixture-of-experts model load on 32GB-class hardware, but it does not make that model the sensible default for a coding agent. Treat it as a way to increase model capacity, not as a speed fix. For long-output interactive agents, a smaller model that fits fully in memory, whether dense-architecture or heavily quantized, remains the better starting point; streamed MoEs are more plausible for short outputs or batch jobs, provided a trial on your own hardware shows acceptable throughput.
The performance figures come from one Hugging Face community post: the author’s own measurements plus one result he cites from another post. No independent benchmark is cited for these results. The two headline speed results concern the same model, Qwen3.8-Flash-Next, measured on two different machines. Between them they demonstrate the capacity feat on a 32GB Mac; neither demonstrates it at a generation rate the author considers usable for coding agents.
The useful distinction is therefore between fitting the weights and feeding them fast enough. Expert streaming addresses the first problem. The second becomes a memory-bandwidth problem, and that is where the biggest-model-that-fits strategy can fail.
Active parameters do not determine the memory bill
A mixture-of-experts model activates only part of its parameters for each token. That can make the active-parameter count look reassuringly small, but it does not reduce the total number of weights that must be stored somewhere.
The MoE-CORE preprint supplies author-reported examples of the gap:
| Model | Total parameters | Active parameters per token |
|---|---|---|
| DeepSeek-V4-Flash | 284B | 13B |
| DeepSeek-V4-Pro | 1.6T | 49B |
| GLM-5.2 | 744B | 40B |
| Kimi K3 | 2.8T | 104B |
DeepSeek-V4-Flash activating 13B parameters per token does not make its storage requirement equivalent to that of a 13B dense model. The other experts still exist and need a home, whether that is VRAM, system RAM or disk.
The FORGE preprint independently supports why streaming is nevertheless possible. Its authors report that expert weights account for 94–98% of the parameter footprint across five frontier MoEs. That closely corroborates the community post’s claim that experts make up “90%+” of a MoE’s weights. Most of the model is therefore eligible to live outside the GPU, while a smaller non-expert portion and the currently needed experts remain closer to the compute.
FORGE also reports FP8 weight footprints ranging from 31GB for Qwen3-30B to 742GB for GLM-5. Those are weight footprints, not guarantees that the complete runtime will fit in the same amount of memory. They show the real capacity spread: some MoEs are plausibly local with careful memory management, while others remain far beyond a consumer machine even before generation speed enters the discussion.
What expert streaming changes
The community post describes Strata, its reference streaming engine, this way: “Only the hot experts stay in VRAM, the rest goes to RAM, or even to disk.” Hot experts are the ones being kept resident because they are currently useful; the rest are paged in as needed.
This reduces the amount of memory that must be resident on the GPU. It does not reduce the model’s total footprint, and it does not eliminate the cost of moving nonresident experts. When generation needs an expert that is not in VRAM, the system must fetch it across the RAM or storage path.
The bandwidth ceiling on the streamed Mac configurations is an inference from that mechanism, not a quantity measured in any of the cited Mac results. Token generation can become limited by how quickly the machine delivers experts rather than by whether the complete model can be loaded. Cache behavior and expert locality can reduce that traffic, and the MoE-CORE preprint measures both: on DeepSeek-V4-Flash, raising the number of cached experts per layer from 12 to 120 lifts the unweighted mean exact hit rate from 47.47% to 96.43%, and its ablations isolate the effects of cache replacement, domain-informed residency and prefetch depth on time per output token. Those measurements come from Ascend NPU appliances with host-memory offloading, however. None of the cited results measures cache behavior or expert locality on the streamed Mac setups; the closest the post comes is the author’s remark that no cache policy beat LRU by much.
That distinction changes the purchasing and model-selection question. “Will this model fit?” becomes two questions:
- Is there enough combined storage and memory for the total weight footprint and runtime?
- Can the machine supply the required experts quickly enough for the way I generate text?
A configuration can pass the first test and fail the second.
Two speed figures, two different machines
The first reported result comes from the author’s own work machine, a 32GB MacBook M5, running his Transformers expert-streaming implementation of Qwen3.8-Flash-Next, the model behind the post’s 180B headline. The capacity half is concrete: after loading, it uses ~18 GiB, and a 6K prompt peaks at 25.4 GiB, which he describes as about the limit of the machine. The speed half is his final benchmark: 13.9 tok/s decode for Transformers against Strata’s 14.8 tok/s on the same machine with the same prompts, or in his words, “about 6% behind Strata on short decode, with slightly faster prefill, all through Transformers”.
That is a useful comparison between two streaming implementations on identical hardware. It does not show streaming coming within 6% of a dense model running fully in VRAM; no such comparison appears in the post.
The second result concerns the same model on a different machine. The author cites a separate post reporting Qwen3.8-Flash-Next at 13.5 tok/s on a 32GB Mac M4, faster than llama.cpp (community post). His response is the decision-relevant part: “That’s too slow for how I use coding agents.”
Read together, the two results corroborate each other: same model, similar 32GB Mac hardware, generation around 13.5–14 tok/s. What they do not provide is a 32GB configuration judged usable for coding agents. The only verdict on record is the author’s, and it is negative: his conclusion about the M5 run reads, “It runs at ~14 tok/s, close to Strata, and it’s still too slow for how I use coding agents.” For the coding work, he stays on his home desktop, a dual-RTX-4090 machine running Qwen 3.8 27B.
What 13.5 tok/s means depends on the workload
A generation rate is not a universal usability score. The workload determines how long the user is exposed to it.
Long-output coding agents are the clearest casualty in these results. Agentic coding often involves extended explanations, patches, tool arguments and multi-step responses. A per-token generation cost is incurred throughout those outputs. The author’s rejection of the 13.5–14 tok/s rates, both the cited M4 post and his own M5 result, is specifically grounded in this usage, so it should not be generalized into a claim that the rate is unusable for everything. It is directly relevant to readers whose agents produce long responses while a developer waits.
Autocomplete and other short-output interactions limit the number of generated tokens, so a slow sustained generation rate may matter less to total waiting time. That is a conditional inference, not a reported autocomplete benchmark. No cited autocomplete latency threshold or editor measurement shows that 13.5 tok/s feels responsive. Short output reduces exposure to the rate; it does not prove acceptable interactivity.
Batch work offers the strongest case for trying a streamed MoE because jobs can be allowed to finish without someone waiting at each turn. But batch viability still depends on total throughput and completion time, neither of which is benchmarked here. The correct conclusion is that batch work deserves a trial, not that streaming is automatically economical for it.
This is why a single tokens-per-second figure cannot settle the model choice. The same reported rate was rejected for one workload while remaining unmeasured for the others.
The documented recipe assumes 64GB, not 32GB
Strata’s hardware guidance adds a counterweight to the headline, but it is not vendor documentation. The Strata community site describes itself as an “Unofficial Community Documentation & Quickstart Hub,” and its recipe for running Qwen3.8-Flash-Next 125B (the community site’s label for the model the Hugging Face post headlines at 180B) on a single 12GB NVIDIA graphics card comes with 64GB of system RAM.
The GPU number is easy to quote without the accompanying system-memory requirement. Both matter, because the documented recipe uses twice the headline article’s 32GB RAM budget. The site does publish speed figures: an empirical 21.4 tok/s for Strata on an RTX 4070 with 64GB RAM, plus estimated ranges of 12–18, 25–38 and 45–65 tok/s across its hardware tiers. None of these establishes coding-agent usability. The 21.4 tok/s result is a single community-published engine comparison on hardware unrelated to the Mac runs, and the ranges are estimates attached to tiers that all assume 64GB or more of system RAM. It is a community-described capacity configuration, not evidence that the same model class is practical with only 32GB.
Match the model to the workload, not just the memory budget
| Setup or option | What the results establish | Practical decision |
|---|---|---|
| 32GB Mac M4 running Qwen3.8-Flash-Next | A separately cited post reports 13.5 tok/s, faster than llama.cpp, and the author rejects that rate for coding agents | Do not select it for long-output interactive agents on capacity alone. Short-output or batch suitability remains unmeasured |
| 32GB MacBook M5, Transformers expert streaming | The author’s own run: 13.9 tok/s decode versus Strata’s 14.8 on the same machine and prompts, judged too slow for coding agents | A proven capacity feat at a rejected speed, and a useful reference point for trials. No dense-model comparison is reported |
| 12GB VRAM plus 64GB system RAM | An unofficial community site documents Qwen3.8-Flash-Next 125B through Strata, with an empirical 21.4 tok/s on an RTX 4070 and 64GB RAM | Treat it as a community recipe requiring validation, not as a speed guarantee or a 32GB recommendation |
| Streamed 180B-class MoE on 32GB-class hardware | The community post supports the capacity premise with its own M5 run; the measured 32GB rates, 13.5 and about 14 tok/s, were rejected by the author for agent work | Prefer it only when the larger model serves a real workload need and your own generation test passes |
| Smaller model, ~27B class (format unspecified in the post) | The author’s own arrangement: Qwen 3.8 27B on Orthrus, his dual-RTX-4090 desktop, handles his coding tasks; the post does not specify that model’s format. No speed baseline or measured crossover against a streamed MoE is cited | The safer starting point for interactive agents if it fits your runtime, because it avoids making expert paging the central dependency. Benchmark it rather than assuming a speed margin |
The recommendation toward the smaller-model option is directional. No generation benchmark for the author’s model is cited, no coding-quality comparison exists, and no measured crossover shows where its speed advantage would outweigh any capability difference from a streamed MoE. What the cited results do show is that fitting a much larger MoE is insufficient, and that the cited Mac generation rates were explicitly rejected for the author’s agent workload.
Where the tradeoff breaks
I would start an interactive coding-agent setup with a smaller model that fits fully in memory, whether dense-architecture or heavily quantized, then test a streamed MoE only if there is a concrete task the smaller model cannot handle adequately. That keeps the default aligned with the workload instead of treating total parameter count as the objective. The author arrived at the same place from the other direction: after building the streaming implementation, the desktop running his Qwen 3.8 27B keeps doing the real work. If the work is short-output or batch-oriented, the priority reverses: a streamed MoE becomes worth trialing on a machine with a fast VRAM, RAM and SSD path, with completion time measured on representative jobs.
Several implementation questions remain open. Nothing cited here establishes Strata’s current feature set, the merged status of the Transformers expert-streaming pull request, or present availability of the converted Qwen3.8-Flash-Next weights. Those should be verified against the exact software version and model files before planning a migration.
The durable rule is simpler than any of those release details: total parameters determine what must be stored; the expert-delivery path helps determine whether generation is usable. Expert streaming can remove a capacity wall, but for long-output coding agents it can replace that wall with a bandwidth ceiling.
Frequently Asked Questions
How much memory does the 180B model use on a 32GB Mac?
after loading, it uses ~18 GiB, and a 6K prompt peaks at 25.4 GiB
What is the documented RAM requirement for the Strata recipe?
its recipe for running Qwen3.8-Flash-Next 125B (the community site’s label for the model the Hugging Face post headlines at 180B) on a single 12GB NVIDIA graphics card comes with 64GB of system RAM.
Join the discussion
Share a useful perspective or ask a question about this article.