Why local Kimi K3 throughput is set by storage, not compute
Moonshot frames K3 as efficient-for-its-class, citing roughly 2.5x the overall scaling efficiency of K2 in its K3 quickstart documentation. Those claims describe training and serving economics, not desktop throughput. None of the vendor pages speak to running K3 on consumer hardware, and the reason is structural: the throughput a local run sees is set by storage latency, not by the accelerator or the active-compute profile the vendor is optimizing.
The full weights only became available on July 27, 2026, per the quickstart page, so any local measurement from the weeks after launch runs freshly shipped weights against tooling (vLLM, llama.cpp, GGUF) with immature support for a trillion-parameter mixture-of-experts. The exact quantization, the CPU-to-NVMe offload split, and whether every expert shard was resident all vary by configuration, and any reported throughput could move by multiples in either direction depending on those choices. Treat any single local-throughput figure as a directional reading, not a spec line, until it is replicated with a known quant and a documented offload config.
Why 2.8 trillion parameters don’t fit in consumer RAM
Whatever quantization a local run uses, the arithmetic says consumer RAM cannot hold Kimi K3’s full weight footprint at any precision a practitioner would call usable.
K3 carries 2.8 trillion parameters, and for a mixture-of-experts model the total parameter count is the residency bill. Every expert’s weights must be stored, even though only 16 of 896 fire per token. At 8-bit, that is roughly 2.8 TB of weights. At 4-bit, a common quality floor for local runs, it is still about 1.4 TB. At 2-bit, roughly 700 GB. To compress the whole model into roughly 32 GB of consumer RAM you would have to land below 0.1 bits per parameter, beneath any quantization scheme that preserves usable output. The configuration is not “K3 in RAM”; it is “a shard of K3 in RAM with the rest on disk.”
That reshapes what a local run is actually measuring. Each generated token activates 16 experts, and whichever of those 16 are not resident get fetched from storage on demand. With roughly 98 percent of the model’s weights offloaded, most expert lookups are NVMe reads, and the run becomes bound by random-read storage latency rather than by GPU compute or attention. Random is the operative word. The router decides which experts fire from the input at inference time, so the sequence of weight fetches is input-dependent and cannot be laid out for sequential prefetch. NVMe delivers on sequential reads and falls off on random ones, and expert paging is the random case. A configuration that keeps most of its weights off-RAM pays random-read latency on most tokens, which is why the throughput floor sits so low instead of tracking GPU FLOPS. Adding a faster accelerator does not move it; the bottleneck runs between the SSD and memory, not on the accelerator. Throughput does not collapse because the model is slow; it collapses because the weights are not where the silicon is.
The input-dependent routing also defeats the obvious caching play. You cannot prefetch the experts a token will need, because the model decides which experts a token needs by running the router on that token’s hidden state, and that state is only available after the preceding tokens have been computed. Every expert fetch is therefore demand-driven and synchronous with the forward pass. A hot expert that fires on common tokens may stay resident if the offloader is clever about eviction, but the long tail of experts means most lookups are cold. Workstation NVMe is built for sequential throughput and for the small-block random I/O an operating system generates, not for whole-expert shards pulled on demand in the middle of a forward pass. The storage subsystem is asked to do the one thing it is worst at, on most tokens, and the throughput floor reflects that.
This is the mechanism any local-throughput figure is sitting on top of. The floor is set by resident storage, not by sparse active compute, and storage is the dimension consumer hardware loses by two orders of magnitude.
How MoE sparsity changes the bottleneck without lowering the floor
Sparsity is why K3 is cheap to serve from Moonshot’s infrastructure and slow to serve from yours: it cuts active FLOPs per token but leaves the full 2.8-trillion-parameter footprint resident.
Activating 16 of 896 experts means roughly 1.8 percent of expert parameters compute on any given token. That is genuinely efficient for active FLOPs, which is the basis for Moonshot’s claim of about 2.5x scaling efficiency over K2. The company attributes that gain to Kimi Delta Attention, Attention Residuals, and the Stable LatentMoE framework. On a serving farm with all 896 experts distributed across GPU memory, the active compute per token is small and tokens are inexpensive.
The error is reading “sparse” as “small.” Every expert must still be addressable on every token, because which 16 fire depends on the input. On a workstation, “addressable” means “resident in memory or a fast page-in away.” With 2.8 trillion parameters, neither option fits inside consumer RAM. Moonshot markets K3 as the first open-source model in the three-trillion-parameter class, and that is a storage milestone first, a compute milestone second.
This is where the local-versus-API playbook that works for 7B-13B dense models stops working. Dense models in that band fit their entire footprint in consumer VRAM, so local throughput is compute-bound and can be competitive with an API for small loads. The same logic applied to a frontier MoE is a category error. The relevant constraint has moved from active compute to memory residency, and consumer memory loses that contest badly. The privacy and cost tradeoff that local execution is supposed to buy you gets pushed back onto whoever holds the serving contract, because there is no second option at the workstation.
The category error runs through the whole cost model. The mental model a lot of practitioners carry from running local dense models has three line items: VRAM for weights, VRAM for the KV cache, and compute for token generation. For a dense model in that band all three fit inside a single consumer GPU, and the bottleneck is whichever line item you skimped on. For a frontier MoE of this scale the weight term dominates by orders of magnitude and never enters VRAM at all, which leaves the other two irrelevant. Adding VRAM for a larger KV cache, or a faster GPU to compute the active experts, addresses costs that are not on the critical path. The upgrade that would actually move the floor is more fast RAM or faster storage, and the amounts involved put it outside the consumer bracket entirely.
When to run Kimi K3 locally and when to call the API
The routing rule is whether a human is waiting. With most of the model’s weights paging from NVMe, per-token latency sits far above what an interactive loop can tolerate. No developer sits through a chat reply at that pace, and no agentic loop that iterates on model output finishes in useful time.
Every interactive workload lands on the API side of the line. Chat, copilot-style coding, and agent tool loops all depend on the model returning tokens faster than a person loses patience, and K3 on consumer RAM cannot. These workloads must run on infrastructure that actually holds the weights.
The only local candidates are workloads with nobody watching. Offline evaluation passes, dataset scoring, and batch generation that you start before leaving for the day and read the next morning can absorb a thousandfold slower rate, because the per-token latency amortizes across thousands of examples. Even there the economics are questionable unless you already own the hardware and have a reason not to send the data to Moonshot.
A workstation sized to hold even a 2-bit quantization of K3 is not a consumer machine; it is a server with hundreds of gigabytes of RAM and multiple NVMe drives, and the amortization only works if that hardware is already sunk cost. Against the API’s per-token pricing, an overnight batch has to clear a high bar of data sensitivity before the local option wins on price. Local batch is a compliance tool, not a cost tool: you run it locally because the data cannot leave, not because the bill is lower.
The table below maps the decision.
| Workload | Who is waiting? | Viable locally? | Route to |
|---|---|---|---|
| Interactive chat | User, each turn | No | Kimi API |
| Copilot-style coding | Developer, each suggestion | No | Kimi API |
| Agentic tool loops | System, each step | No | Kimi API |
| Batch evaluation / scoring | Nobody (overnight) | Possibly | Local, only if data must stay on-prem |
| Offline dataset passes | Nobody (overnight) | Possibly | Local, only if hardware is already owned |
The line is not subtle. If the failure mode is “a developer watching a cursor,” the answer is the API. If the failure mode is “this finishes by morning,” local is on the table, contingent on storage budget and an honest price comparison against the API.
What the Kimi API surface gives you that local cannot
Routing to the API buys the thing consumer hardware cannot provide at interactive speed: infrastructure that actually holds all 2.8 trillion parameters resident. K3 also exposes a native 1M-token context window and native visual understanding on the API, neither of which a paging-bound local run can usefully exercise.
The API is the only practical surface for any workload that needs K3’s full capability at interactive latency. A local run would let you control decoding settings and keep data on-prem, but the residency math makes that option academic for anything a human waits on.
Which Moonshot model actually runs on consumer hardware
For teams whose hard requirement is local execution, the right Moonshot model is not K3 but Kimi K2.6, the company’s smaller open model for coding and agent workloads.
K2.6 is positioned as the open-source coding and agent model in Moonshot’s lineup, a role that implies a footprint designed to run where K3 cannot. K2.6’s parameter count is not documented in the available sources, but the positioning is the signal: K2.6 is the model you reach for when the constraint is “runs on my machine,” and K3 is the model you reach for when the constraint is “frontier capability, served by someone who owns enough HBM.”
The mistake to avoid is treating K3’s open weights as a local-deployment promise. Open weights for a 2.8-trillion-parameter MoE mean you can audit the model, fine-tune it, or serve it on adequate infrastructure. They do not mean it runs on a workstation. If the use case is local coding assistance, K2.6 is the honest answer. K3 behind the API is the capability upgrade you take once local execution is no longer the binding constraint.
The verdict
The practical call is to route every interactive Kimi K3 workload to the API and reserve local runs for batch jobs you can leave running overnight.
The durable conclusion does not depend on any single throughput measurement. 2.8 trillion parameters do not fit in consumer RAM at usable precision, so any local configuration is bound by weight paging, and any interactive workload is non-viable. The routing split and the breakdown of the dense-model local-versus-API playbook follow from the architecture, not from a benchmark.
What does shift over time is the precise floor. The specific throughput will move as vLLM, llama.cpp, and GGUF tooling for trillion-parameter MoE matures, and a better offload strategy or a tuned quant could move the rate by multiples in either direction. If you need a number for capacity planning, measure it yourself with a documented quantization and offload configuration before it enters any planning document. The architectural insight is the part worth carrying forward. The exact rate is the part worth re-measuring.
Frequently Asked Questions
Does Kimi K3’s native 1M context window work with local offloading?
No. The 1M token context requires a KV cache that consumes roughly 16 GB of VRAM at 16-bit precision, which is unavailable on the consumer hardware already struggling with the model’s weights. The API handles this capacity in its high-bandwidth memory, while local runs are limited by the RAM available for the 2.8T weight footprint.
How does Kimi K3’s reasoning_effort parameter differ from local decoding?
The API exposes a reasoning_effort parameter with low, high, and max settings that dynamically adjusts internal compute steps, a control absent from local runtimes. Local operators are restricted to fixed sampling parameters like temperature=1.0 and top_p=0.95, meaning they cannot trade latency for deeper reasoning chains the way the API allows.
What is the actual VRAM floor for running Kimi K3 locally?
Even at 4-bit quantization, the 2.8 trillion parameters require roughly 1.4 TB of storage, forcing nearly all weights onto NVMe. This creates a random-read bottleneck that makes interactive throughput impossible, as the system must fetch expert shards on demand rather than accessing resident memory.
Is Kimi K2.6 a viable alternative for local coding workloads?
Yes. Kimi K2.6 is positioned as the coding and agent model in Moonshot’s lineup, with a parameter count small enough to fit in consumer VRAM. Unlike K3, K2.6 does not suffer from the trillion-parameter residency floor, making it the practical choice for local execution.