How Much VRAM Do You Need to Fine-Tune a MoE LLM?

The short answer: for a full fine-tune of a sparse MoE model, budget roughly 12 bytes per trainable parameter before you count a single activation, 2 bytes of bf16 weight, 2 bytes of bf16 gradient, and 8 bytes of AdamW state per parameter, and remember that in a MoE checkpoint almost every parameter is an expert parameter, even though each token only activates a few. The practical consequence is that the model’s total parameter count, not its active-per-token count, sets your VRAM bill, and the bill lands long before compute becomes the constraint. The decision that matters is which lever shrinks that ledger to fit your hardware: LoRA on selected modules, full training with reduced-precision or offloaded optimizer state, or a newer in-backward update method. This article builds that ledger with inputs you can rerun on your own checkpoint. Where figures come from the underlying papers, they are author-reported, observed on 2026-11-06, and one of them, the in-backward method, is a single November 2026 preprint with no independent verification. Those caveats are part of the answer, not a footnote to it.

The ledger: bytes per parameter for weights, gradients, and AdamW state

Fine-tuning memory decomposes into a small number of per-parameter terms. Each trainable parameter needs:

• Weight storage: 2 bytes at bf16, the standard training precision, or roughly 1 byte at fp8.
• Gradient storage: 2 bytes per parameter at bf16, held for every parameter that receives an update.
• Optimizer state: for AdamW, two float32 moments per parameter. As the tiered optimizer-state paper puts it, AdamW “keeps two float32 moments for every parameter, 8 bytes of state shadowing every 2-byte bfloat16 weight.”

That optimizer line is the one that kills naive budget estimates. The 8 bytes of fp32 moments are four times the weight bytes alone, so a full AdamW fine-tune carries approximately 12 bytes of persistent state per trainable parameter before activations, temporary buffers, or framework overhead enter the picture. Activations scale with batch size and sequence length and can be traded away with checkpointing; weights, gradients, and optimizer moments are harder to shrink without changing the method.

There is also a subtlety in when gradient memory is live. The Sonar-MoE preprint describes the pattern directly: “A standard training step runs the whole backward pass and only then calls the optimizer, so the gradient of the last MoE layer waits in memory until the backward pass has reached the first layer.” Every expert gradient in the model is simultaneously resident at the end of the backward pass. The paper notes why this wait is unnecessary in principle: “An element-wise optimizer such as AdamW (Loshchilov and Hutter, 2019) does not need this wait, since the update of a tensor depends only on that tensor’s gradient and optimizer state.” Keep that observation in mind; it becomes one of the levers below.

Why MoE breaks dense-model math

Dense-model VRAM rules of thumb, the familiar “16 bytes per parameter” style formulas, at least point at the right number of parameters. For MoE they fail in the other direction, in a way that surprises people the first time they budget a run: routing is an inference-time property, not a training-memory property. Each token activates a small subset of experts, but backpropagation produces gradients for every expert that was activated anywhere in the batch, and AdamW keeps state for every parameter you choose to train. Unless you freeze experts deliberately, the total parameter count is the one that matters.

And in current MoE checkpoints, the experts are the parameter count. According to the FORGE paper, “At FP8 precision, MoE parameters account for 94–98% of total model size, ranging from 29 GB (Qwen3-30B (Qwen Team, 2025), 128 experts) to 728 GB (GLM-5, 256 experts across 75 MoE layers).” That range is author-reported, and fp8 is an inference-oriented precision rather than the bf16 you would typically train in, but the structural point survives any precision conversion: attention, embeddings, and shared layers are a rounding error in the memory ledger. When you size a MoE fine-tune, you are sizing the experts.

One boundary worth stating: the quoted range names Qwen3-30B and GLM-5 at its ends, and no DeepSeek or Kimi checkpoint size appears in the papers collected here, so this article does not quote figures for those families. The 94–98% dominance argument covers the surveyed class; apply the worksheet below to your own checkpoint’s config rather than extrapolating the named examples.

Worksheet walkthrough: OLMoE-1B-7B

The Sonar-MoE paper supplies a fully worked example on OLMoE-1B-7B, a small research MoE that is convenient precisely because the numbers are checkable by hand: “In OLMoE-1B-7B (Muennighoff et al., 2024), for example, the experts hold 6.44B of the 6.92B parameters, and their bf16 gradients alone take 12.9 GB.” For the OLMoE configuration (L=16, E=64, d=2048, n=1024) the expert gradients total 6.44B elements, which is 12.9 GB at 2 bytes per element.

Run the full ledger for a full fine-tune of all parameters on that checkpoint:

About 83 GB of persistent state, on a model marketed as “1B active / 7B total,” before activations, before temporary buffers, before any master fp32 weight copy your trainer may keep. That is already past a single 80 GB card once overheads arrive, for a checkpoint at the small end of the Qwen/GLM class the FORGE paper surveys. The same multiplication applied to a checkpoint with hundreds of billions of total parameters explains why single-node full fine-tuning of frontier-class MoE models is not a budget question but an architecture question.

The useful property of this table is that every input is exposed. Pull the trainable parameter count from your checkpoint’s config, multiply by 2, 2, and 8, and you have the floor of your memory budget. What the table cannot give you is a measured peak for a specific distributed recipe. None of the papers cited here reports end-to-end VRAM totals for ZeRO-3, FSDP, or CPU/NVMe offload configurations on current checkpoints, so treat any sharded-total estimate as arithmetic you must validate on your trainer, not a cited result.

Lever 1: LoRA on selected modules

Low-rank adapters shrink two of the three ledger rows at once: the frozen base weights produce no gradients and no AdamW state, so the 10-bytes-per-parameter gradient-plus-optimizer burden applies only to the adapter parameters, which are a small fraction of the total. The cost is compute and expressiveness. On compute, the DR-LoRA paper reports a MoE-specific figure worth knowing: because “the base expert computation dominates total FLOPs (98.5%), the overall increase over the no-LoRA baseline is only 3.1%.” That is an author-reported analysis for the OLMoE architecture, but the underlying reason generalizes: in a MoE, the frozen expert matmuls are so large relative to the adapter terms that the adapters are nearly free in FLOPs. You are buying most of the memory relief at a few percent of extra compute.

The expressiveness trade is the real decision. LoRA constrains the update to a low-rank subspace of whichever modules you attach adapters to. If your task requires moving the expert behavior substantially, new domain tokenization patterns, large style shifts, specialist capability, that constraint may bind, and this article’s evidence cannot tell you where your task’s threshold sits. What the arithmetic does say: LoRA is the lever that changes the memory bill from “total parameters” to “adapter parameters,” and it is the only lever here whose semantics match standard training exactly, because it changes what is trained rather than how the update is computed.

Lever 2: full fine-tuning with 8-bit optimizers, and the per-tensor cliff

The second lever keeps full-rank updates but compresses the optimizer row: 8-bit AdamW implementations cut the 8 bytes of fp32 moments to roughly 2 bytes per parameter, which would take the ledger from ~12 to ~6 bytes per trainable parameter. On the OLMoE worksheet above, that is the difference between ~83 GB and ~41.5 GB of persistent state, the difference between needing a second GPU and not. Step time is where you pay it back: the Sonar-MoE benchmarks find that “8-bit AdamW is slower per step (186 vs. 141 ms)” on OLMoE, about 32% slower.

But there is a hard failure mode specific to MoE, and it does not degrade gracefully. The tiered-state paper, citing bitsandbytes issue 1785, reports that current 8-bit optimizer kernels “terminate the process, a hard exit from C++ rather than an exception, once a single parameter tensor reaches” 2^31 elements, “a size that fused expert weights reach in larger MoE models.” Many MoE implementations store each layer’s experts as one fused tensor for efficient grouped matmuls; once that fused tensor crosses roughly 2.15 billion elements, the kernel does not throw a catchable error, the process dies. A run can pass its first-step smoke test and still be doomed if you tested on a configuration below the threshold.

So the pre-flight check is mechanical: compute the element count of the largest single fused expert tensor in your checkpoint (experts per layer times per-expert parameter count, per layer) and compare it against 2^31. If you are near or over the line, options include sharding the expert tensor differently, using paged or CPU-offloaded optimizer states instead, or accepting fp32 moments with ZeRO-style sharding. None of those recipes comes with measured totals in the papers collected here, so validate them on your stack. What the evidence does establish is that “just use 8-bit Adam” is not a safe default for fused-expert checkpoints.

Lever 3: in-backward expert updates, labeled author-reported

The third lever is the newest and the least verified. The Sonar-MoE preprint, posted in November 2026, proposes computing expert optimizer updates inside the backward pass, as each expert tensor’s gradient becomes available, rather than holding all expert gradients until backward completes. Mechanically, this exploits the element-wise property quoted earlier: AdamW’s update for a tensor depends only on that tensor’s gradient and its own moments, so there is no mathematical reason to wait. The peak gradient memory collapses from “all expert gradients at once” to “the gradients of the layers currently in flight,” which attacks exactly the 12.9 GB term in the OLMoE worksheet.

I would treat every saving figure from this paper as author-reported and unreviewed, and the paper itself hands you the reason. Its equivalence claim is bounded: “Without gradient accumulation, Sonar-MoE performs the same arithmetic as standard training that clips only the non-expert gradients.” Two distinct caveats live in that sentence, and both matter operationally.

Gradient accumulation. The claimed equivalence holds only when each step is a single micro-batch. Most memory-constrained fine-tuning recipes accumulate gradients over several microbatches precisely because memory is tight, the exact population this method targets. The preprint does not leave that case open; it works it out and measures it. Section 4.4 derives what happens: “A post-accumulate hook fires as soon as each microbatch’s gradient is ready, so the hook schedule applies one AdamW update per microbatch instead. These are not equivalent to a single update using the combined gradients, because both the weights and AdamW’s optimizer state change between microbatches.” Section 5.6 measures the price: with accumulation the method “runs the optimizer A times per step and is slower than standard training with global clipping, by 5% at A=2, 23% at A=4 and 35% at A=8, while still saving 12.9 GB.” The 12.9 GB memory saving therefore does transfer to accumulated runs. What does not transfer is the arithmetic or the speed: under accumulation Sonar-MoE is a different and slower optimizer, which is why the paper confines its headline results to A=1.

Gradient clipping. Standard practice clips the global gradient norm across all parameters. Under the in-backward scheme, the paper’s equivalence is to training “that clips only the non-expert gradients,” because expert gradients are consumed by the optimizer as they arrive, before a global norm could be computed over them. Whether losing expert-side clipping affects stability for your task is an empirical question the preprint cannot answer for you.

Neither caveat makes the method wrong; both make it a different training recipe whose savings come with semantics attached. The adoption question shifts from “do I have enough GPUs” to “do I trust this trainer implementation to handle accumulation and clipping the way my recipe expects.” That is a real trade, and for some teams it is a good one, trainer-implementation risk on one node can be cheaper than a second node. But it is risk, and it is currently documented by exactly one unreviewed preprint.

Comparing the levers

The levers are not mutually exclusive, LoRA plus an 8-bit optimizer stacks, for instance, but each one you add past the first is another place where the recipe you ran diverges from the recipe you can cite.

The pre-rental checklist

Before committing money to hardware, work through the ledger in order:

1. Count trainable parameters from the config. Total, not active. Expect experts to be nearly all of it: 94–98% of checkpoint size at fp8 in the FORGE survey’s model class, author-reported.
2. Multiply. 2 bytes weights + 2 bytes gradients + 8 bytes AdamW moments per parameter gives the full fine-tuning floor. Compare with your target hardware after reserving room for activations and framework overhead.
3. If over budget, pick a lever deliberately. LoRA if a constrained update is acceptable; 8-bit optimizer state if every fused expert tensor stays under 2^31 elements; in-backward updates only if your recipe does not rely on gradient accumulation and you accept non-expert-only clipping.
4. Validate before scaling. Run one step at target batch size and log actual peak memory. The worksheet is a floor computed from per-parameter arithmetic, not a measured ceiling for any specific trainer.

The evidence cannot settle everything here. The MoE magnitudes and the method-level cost figures are author-reported; DeepSeek- and Kimi-class checkpoint sizes appear nowhere in the papers quoted; and none of them measures real ZeRO-3 or offload configurations end to end. Those are gaps, not estimates I am willing to invent. What is solid, because it is arithmetic on documented byte costs, is the ledger itself: on a MoE checkpoint, the expert parameters you never activate per token are still the parameters you pay to train, and the difference between a run that fits and a run that dies at the optimizer step is usually visible on paper before you rent anything.

Frequently Asked Questions

How many bytes of persistent state does a full AdamW fine-tune require per parameter?

That optimizer line is the one that kills naive budget estimates. The 8 bytes of fp32 moments are four times the weight bytes alone, so a full AdamW fine-tune carries approximately 12 bytes of persistent state per trainable parameter before activations, temporary buffers, or framework overhead enter the picture.

What is the hard failure mode for 8-bit optimizers in MoE models?

The tiered-state paper, citing bitsandbytes issue 1785, reports that current 8-bit optimizer kernels “terminate the process, a hard exit from C++ rather than an exception, once a single parameter tensor reaches” 2^31 elements, “a size that fused expert weights reach in larger MoE models.”

Does the in-backward update method work with gradient accumulation?

Section 4.4 derives what happens: “A post-accumulate hook fires as soon as each microbatch’s gradient is ready, so the hook schedule applies one AdamW update per microbatch instead. These are not equivalent to a single update using the combined gradients, because both the weights and AdamW’s optimizer state change between microbatches.”