Open-weight mixture-of-experts models ship total footprints no workstation holds: Llama 4 Scout, 109B total parameters per Ollama’s card, is still about 55GB at 4-bit quantisation, over a 48GB Mac’s ceiling before the OS and KV cache take their share. Expert streaming, paging expert weights from SSD as the router demands them instead of keeping the model resident, is the technique that would close that gap on hardware you already own. No replicated public benchmark of it existed as of September 2026, so what follows is conditional analysis. If the arithmetic holds in practice, unified-memory capacity stops deciding which models fit locally, and storage bandwidth decides how fast they run. That changes what local-first teams should buy.
Why do so many models no longer fit in workstation memory?
Mixture-of-experts architectures decouple total parameter count from per-token compute, so open-weight models now ship total footprints far beyond workstation RAM while advertising per-token costs that sound manageable.
In the dense-model era, the fit question was linear: parameters times bytes per weight, checked against your memory budget. MoE broke that arithmetic. A router now selects a small subset of “expert” feed-forward blocks per token, which means inference cost scales with the active slice while the storage and residency requirement scales with the whole thing. Meta’s Llama 4 generation is the cleanest public reference: Ollama’s model card lists Scout at 109B total parameters with 17B active, and Maverick at 400B total with the same 17B active. Both are multimodal MoE models with a 12-language roster.
The second-order effect is that the capacity gap is structural rather than accidental. Vendors are not shipping bloated models; they are trading residency for per-token compute, because per-token compute is what serving costs. For a local-first team, that trade inverts. You are not paying per token at datacenter rates. You are paying once, in hardware, for the whole 109B or 400B to live somewhere addressable. Capacity planning shifts from “which quantisation fits” to “which parts of this model actually need to be fast.”
What does “109B total, 17B active” mean for a RAM budget?
Every parameter must remain addressable even though only the active slice computes for a given token, so memory planning is driven by total bytes at your chosen quantisation, not by the active-parameter number in the marketing.
The mechanism: a gating network inspects each token and routes it to a handful of experts. The attention blocks and shared layers participate in every token; the expert weights, which make up most of the parameter count, participate only when routed to. On Scout’s numbers, the active set is roughly 16% of the total, 17B of 109B per Ollama’s card. The other 84% sits idle for any given token but must be available for the next one, because routing changes token by token.
Run the arithmetic. At two bytes per weight, 109B parameters is about 218GB. At 4-bit quantisation it lands near 55GB: still over a 48GB Mac’s ceiling, and the machine needs room for the OS, the KV cache, and the always-on shared layers. The active set at 4-bit, by contrast, is under 10GB. That gap between “must be addressable” and “touched per token” is the entire wedge that expert streaming exploits. (The 17B is the headline number. The 109B is the invoice.)
Maverick makes the same point louder: 400B total at 4-bit is on the order of 200GB, well past any current Mac configuration, while its active set is the same 17B as Scout’s. Per Ollama’s Llama 4 page, both models route identically in that respect. If you can stream experts fast enough, the difference between running Scout and running Maverick locally becomes a storage question, not a memory question.
How do quantisation, mmap offload, and expert streaming differ?
Three techniques relax the capacity constraint, and they stack: quantisation shrinks each weight, mmap-style offload pages the whole model through memory on demand, and expert streaming fetches only the experts each token actually routes to.
Quantisation is the oldest lever and the most orthogonal. It reduces bytes per weight, typically from 16-bit floats to 4-bit or below, and it does not care where the weights live. Its cost is quality, which degrades measurably as you push into aggressive low-bit territory; the trade is well documented and applies identically whether the model is resident or streamed.
mmap-style offload is what llama.cpp does when you load a model larger than RAM: the weight file is memory-mapped and the kernel pages chunks in on demand. This works acceptably when the working set is mostly resident and the miss rate is low. For a fully offloaded dense model it collapses, because every token logically touches the entire weight file, so per-token latency approaches model size divided by storage bandwidth. Even resident local inference is bandwidth-bound, since a decoder reads its whole active working set every token; pushing that working set to SSD makes the same wall much lower and much harder. Dense full-model offload is a novelty mode, useful for extracting one embedding or one completion, not for conversation.
Expert streaming exploits MoE routing instead of fighting it. Because each token only activates a fraction of the experts, the bytes that must move per token drop by the total-to-active ratio before any caching: roughly 6x on Scout’s 109B/17B split, over 20x on Maverick’s 400B/17B. Consecutive tokens often route to overlapping experts, so a modest expert cache in RAM converts bandwidth into hit rate. The binding constraint moves from memory capacity to SSD bandwidth and routing locality, both of which are cheaper to buy than unified memory.
| Approach | What must be resident | Bytes touched per token | Binding constraint | Failure mode |
|---|---|---|---|---|
| Quantisation only | Whole model (smaller) | Whole model | Memory bandwidth | Quality loss at low bit-widths |
| mmap full offload | Shared layers + OS pages | Effectively whole model | Storage bandwidth | Seconds per token on dense models |
| Expert streaming | Shared layers + expert cache | Active experts only | Storage bandwidth, routing locality | Cache misses stall generation |
| Streaming + quantisation | Shared layers + expert cache | Active experts, compressed | Storage bandwidth | Compounds both failure modes, usually tolerably |
Is streamed inference fast enough to use?
There is no replicated throughput number for expert streaming on workstation hardware as of September 2026, so the honest answer is conditional: usability depends less on any single figure than on what fraction of your workload’s wall-clock time is token generation.
Take batch work first. Document summarisation, extraction, classification, and similar pipeline jobs are throughput-oriented and latency-indifferent. Whatever rate streaming sustains, overnight runs convert it into completed work on hardware already owned, with zero per-token spend and no data leaving the building.
Agent loops are the friendliest interactive-adjacent case. An agentic coding or research loop spends most of its wall-clock time on tool calls, test runs, and network waits; generation is a minority of the elapsed time. Dropping generation from resident-inference speed to a streamed rate degrades the loop’s total latency by a fraction, not an order of magnitude.
Interactive chat is where streaming bites. Generation at storage-bound rates leaves no headroom for long generations, and prefill on a long prompt adds a stall before the first token appears. How streaming implementations handle prefill, where routing across hundreds of tokens in parallel may thrash the expert cache, is an open question. For anything a human is watching, a resident quantised model that fits your memory, or a hosted API, remains the sane default.
Long-context single jobs sit in between: tolerable if the job is one-off and the alternative is not running the model at all, painful if repeated.
How much faster is hosted inference?
Hosted wafer-scale inference markets a speed gap that expert streaming cannot close: Cerebras positions its CS-4 system as up to 30x faster than GPU-based inference, and its reported 2025 financials make hosted dependence a defensible architecture choice rather than a bet on a startup’s survival.
The hardware claim, per Cerebras’s product page, is a rack-scale CS-4 built on three WSE-3T wafer-scale processors delivering up to 30x the inference throughput of GPU systems. That is vendor marketing with no workload specifics attached, so treat the multiplier as directional. Even discounting it heavily, hosted wafer-scale serving lands two orders of magnitude above anything a workstation streaming weights from SSD can sustain.
The more interesting fact for a build-vs-buy decision is financial. Cerebras Systems booked $510 million in 2025 revenue with $87 million in net income. A hosted-inference provider that is revenue-generating and profitable is a different dependency risk than a subsidised API burning venture capital; routing production traffic to it is an architecture decision you can defend in a review meeting.
The decision splits cleanly. Interactive, user-facing, latency-sensitive: hosted inference wins by two orders of magnitude and the privacy cost is the price. Batch, background, privacy-sensitive, or cost-capped: local streaming costs electricity and patience, nothing else. The workloads where the choice is genuinely hard, interactive sessions over confidential data, are exactly where resident quantised models that fit your memory remain the answer.
What should local-first teams buy in September 2026?
If capacity stops gating which models run, weight SSD bandwidth over the RAM ceiling in your next purchase; Apple’s Mac mini M6 and M5 Pro availability starting 9.22 makes this a timing question, and llama.cpp’s hardware target list shows the field extends well past Macs.
Apple’s homepage currently advertises Mac mini with M6 and M5 Pro chips, available starting 9.22. That is a teaser, not a spec sheet: configurations, memory ceilings, and SSD options are unconfirmed at the time of writing, and buying advice against a homepage banner is buying advice against marketing. What the timing does mean is that a purchase decision this month is a decision about whether to wait roughly two weeks for the refresh.
The strategic shift matters more than the timing. Under the residency model, the buying heuristic was simple: maximise unified memory, because capacity decided the model tier you could run. Under a working streaming model, capacity is rented from the SSD instead. The marginal dollar moves from RAM to storage bandwidth and to the expert cache that converts that bandwidth into hit rate. Unified memory still matters: shared layers, attention, KV cache, and the OS all live there, and 48GB remains a sensible floor for a 100B-class model. But the difference between 48GB and 64GB buys less than it used to, and the difference between a slow and a fast NVMe path buys more.
The Mac is also not the only target. llama.cpp’s official site lists optimised support for Apple Silicon M Ultra, M Pro, and M Max alongside the NVIDIA RTX 5090, H100, and A100, AMD’s MI300, and the DGX Spark. On discrete-GPU machines, VRAM capacity still gates hard, and expert-streaming tooling outside the unified-memory world is younger. Teams buying GPU rigs for resident inference are making a different, still-legitimate bet: that resident speed on models that fit beats streamed access to models that don’t.
When does expert streaming win?
Expert streaming earns a place for batch and background work on hardware you already own; interactive sessions still belong on resident quantised models or hosted APIs, and nothing should be purchased on the strength of streaming claims until independent replication exists.
The case for streaming is the arithmetic from earlier sections: MoE models ship total footprints 6x to 23x their active sets, the active set is what each token touches, and NVMe bandwidth is cheaper than unified memory. If streaming delivers on that arithmetic, a 48GB Mac stops being a machine that runs 30B-class models and becomes a machine that runs 100B-class models slowly, which for nightly batch work is the same as running them.
The case against is equally concrete. Streaming has no replicated public benchmark: no published accuracy deltas, no prefill characterisation, no variance data across context lengths or models. Cerebras’s 30x speed claim is marketing without workload specifics, even if the direction is obviously correct. Apple’s 9.22 Mac mini availability is a homepage teaser without published configurations. Every load-bearing number in this story is vendor-supplied or derived from vendor-supplied specs.
So the operational verdict: if you own a 48GB-plus Apple Silicon machine and have batch or agentic workloads, expert streaming is worth testing this week, on your own documents, with your own SSD, measuring your own tokens per second. If you are buying hardware, wait for the 9.22 configurations to publish and weight SSD bandwidth alongside memory. If you are building anything interactive, nothing about streaming changes your answer. And when independent replications arrive, they will look like someone else’s machine, someone else’s model, and a number with published methodology.
Frequently Asked Questions
Does expert streaming work on NVIDIA GPUs like the RTX 5090?
Tooling is significantly less mature on discrete GPUs because VRAM capacity still acts as a hard gate, unlike Apple Silicon’s unified memory. While llama.cpp lists the RTX 5090 as an optimized target, the expert-streaming implementations are younger and less tested outside the unified-memory ecosystem, making Macs the safer bet for this specific workflow.
How does expert streaming differ from standard mmap offloading?
mmap pages the entire model file on demand, causing every token to potentially touch the whole weight file and resulting in seconds-per-token latency for dense models. Expert streaming exploits MoE routing to fetch only the specific experts activated per token, reducing the bytes moved per token by the total-to-active ratio, which is roughly 6x for Llama 4 Scout.
What specific hardware metric should I prioritize when buying a Mac for streaming?
Prioritize NVMe storage bandwidth over the unified memory ceiling, as the bottleneck shifts from capacity to data transfer speed once the model is streamed. While 48GB remains a sensible floor for shared layers and KV cache, the marginal performance gain from upgrading to 64GB is smaller than the gain from a faster SSD path for fetching expert weights.
Is Cerebras CS-4 a viable alternative for interactive local inference?
No, Cerebras CS-4 is a rack-scale hosted system, not a local workstation component. It is marketed as up to 30x faster than GPU systems, but it requires a datacenter deployment and represents a hosted dependency rather than a local-first solution, making it unsuitable for teams requiring on-premise data privacy.
