Memory Bandwidth vs FLOPS: What Actually Limits LLM Inference Speed

For interactive LLM serving, tokens per second is set by how fast you move bytes, not by how many FLOPS you bought. During decode, the phase that produces the words a user actually waits on, the full weight matrices are streamed from memory for every single token while the arithmetic per token stays small, so memory bandwidth dominates (arXiv:2512.22066). If you rented GPUs by peak FLOPS and still cannot predict your decode latency, that is the reason: you optimized the wrong ceiling. The practical consequence is that sizing an inference node starts with a bandwidth and capacity question, and FLOPS enters the picture mainly through prefill, the compute-bound half of the workload.

A note on the evidence before the arithmetic: everything in this article rests on eight arXiv preprint versions whose peer-review status was not verified, plus one vendor blog used only as a CPU-side analogy. Where the article moves from reported results to a sizing rule, that step is inference, and its limits are stated where they matter.

The short answer: decode waits on memory, prefill burns FLOPS

Modern LLM inference splits into two phases with opposite bottlenecks. Prefill ingests the prompt in parallel and is heavy on arithmetic; decode generates output one token at a time and is heavy on memory traffic. A recent preprint on prefill and decode bottlenecks states the decode side directly: “The model’s weights are often tens to hundreds of GB, and even though only one token is processed, the full weight matrices must be streamed from memory. Compute per token is relatively small, but memory reads per token are massive and so memory bandwidth dominates” (arXiv:2512.22066). The same paper classifies the other phase: prefill “is significantly more computationally expensive per token than decoding and exhibits quadratic complexity with respect to sequence length,” making it compute-bound (arXiv:2512.22066).

The roofline explanation for why peak FLOPS oversells decode comes from a preprint on KV cache rematerialization: “generating each token only requires performing low intensity matrix-vector multiplications, whereas loading these matrices requires far more memory operations” (arXiv:2508.10395). Arithmetic intensity, the ratio of compute to bytes moved, is what a roofline model uses to decide whether a kernel is limited by the FLOPS ceiling or the bandwidth ceiling. Decode sits far to the bandwidth side of that line for most batch sizes and context length regimes.

This gap is also widening. The same preprint reports the trend often called the Memory Wall directly: “The scaling in peak computational performance is orders of magnitude greater than the corresponding increases in memory capacity and bandwidth on hardware” (arXiv:2508.10395). Each hardware generation has been buying FLOPS faster than it buys bandwidth, so the mismatch between spec-sheet compute and delivered decode speed grows rather than shrinks.

The pattern is not unique to GPUs. A vendor blog profiling CPU database queries found that in a naive GROUP BY, “50.1% of the time, the CPU couldn’t do anything because it was stalling on memory!” (Pivot blog), and the processor retired operations only 16.1% of the time. The backdrop numbers in the same post explain why: a modern CPU executes roughly 4 to 8 instructions per nanosecond, but a RAM access takes roughly 50 to 100 ns (Pivot blog). That is CPU and database evidence, not LLM evidence, so treat it as an analogy only. It is still a useful intuition: when the working set does not fit near the compute, the compute idles.

The ceiling arithmetic: bytes per token divided by effective bandwidth

The sizing rule that follows from the mechanism is simple:

Decode tokens per second ≈ bytes moved per token ÷ effective memory bandwidth

Bytes moved per token is the active model weights plus the KV cache touched for that token. Effective bandwidth is what the memory system sustains under real decode access patterns, not the “up to” number on the spec sheet. Read this as a ceiling: latency overheads, kernel efficiency, and imperfect overlap all push real throughput below it.

Be clear about the epistemic status of this formula. None of the cited papers states it end-to-end or validates it against measured tokens per second across mainstream GPUs. Each component is supported separately: weight streaming dominates decode (arXiv:2512.22066), arithmetic intensity is low (arXiv:2508.10395), and hardware datasheets supply the bandwidth figures cited below. The license to trust spec-based arithmetic at all comes from GenZ, an analytical estimator for LLM inference that its authors “validated against real hardware platforms running various different LLM models, achieving a max geomean error of 5.82%” (arXiv:2406.01698). That is one modeling study, but it is the closest check available here that datasheet math predicts real inference performance within a few percent.

One discipline matters whenever you use the formula: treat quoted bandwidth as a vendor ceiling. “Up to 4 TB/s” is not a sustained decode throughput, and none of the cited studies measures achievable bandwidth against spec. A conservative approach is to compute the ceiling with spec bandwidth, then measure one real decode run on your model and use the ratio as your derating factor.

What counts as bytes: active weights, MoE sparsity, and the KV cache

The numerator is where sizing mistakes hide. Three things change it.

Dense versus MoE activation. For dense models, every parameter moves every token. The GenZ preprint quantifies the contrast: “For LLaMA2-7B, LLaMA3-70B, and GPT3-175B, all model parameters are utilized in each inference iteration, whereas in Mixtral-8x7B and GPT4-1.8T, only 27% and 15% of the total model parameters are activated per inference cycle” (arXiv:2406.01698). Applying dense-model arithmetic to a Mixture-of-Experts model overstates bytes per token by a factor of roughly four to seven, which makes a GPU look far too slow on paper for a model it would actually serve well.

The KV cache grows with context. Decode also reads the accumulated key/value state for the conversation, and that state grows linearly with context length. The arithmetic here is derivable from the cited papers. GenZ gives the fast-memory capacity requirement as model parameters plus a KV cache term, KV cache = 2 × B × (τp + Sb × τd) × H_kv × D/H × N_layers, where B is batch size and the τ terms count prefill and decode tokens (arXiv:2406.01698). In plain terms, each cached token costs roughly 2 × KV heads × head dimension × layers × bytes per element, and the total cache scales linearly with batch size and context length. The XQuant preprint makes the shape concrete for Llama-3.1-8B: with a hidden dimension of 4,096 and four query heads sharing each key/value pair, the key and value activations have dimension 1K each per token, “so the KV cache is of shape l×2K” per layer (arXiv:2508.10395). How much this term matters depends on model size. GenZ reports that for GPT4, moving from QA to RAG-style QA with a tenfold longer context raised the modeled memory-bandwidth requirement by only 8%, which it attributes to the large size of the model itself: when the weight read dominates, context growth is a rounding error (arXiv:2406.01698). For smaller models, longer contexts, or larger batches, the cache term competes harder. The CXL-PNM preprint benchmarks decode specifically at long context, comparing a 1.1 TB/s LPDDR5X platform with 512 GB per module against a DGX appliance with up to eight A100-80GB GPUs at 2 TB/s HBM2E, precisely because KV cache at 1M-token scale exceeds what fits comfortably on a GPU (arXiv:2511.00321). For long conversations, the cache term in the numerator stops being noise and starts competing with the weights term.

Capacity is the gate before bandwidth. If weights plus KV cache do not fit in high-bandwidth memory, you do not get the bandwidth in the first place. Sizing for capacity first, then bandwidth, is not pedantry; it is the order in which the constraints bind.

Bandwidth and capacity across real hardware

The cited studies put concrete numbers on how accelerators differ on the axes that matter for decode. A preprint on KV cache placement in heterogeneous memory describes the GH200 Grace Hopper Superchip as combining “a Hopper GPU with 96 GB of HBM3 delivering up to 4 TB/s of bandwidth, and a Grace CPU connected to 512 GB of LPDDR5X” (arXiv:2508.13231), with the two linked by “NVLink-C2C, offering up to 900 GB/s of bandwidth.” The same paper notes that “LPDDR5X devices can reach speeds of up to 8,533 MT/s, and NVLink 4.0 supports up to 100 GB/s per link” (arXiv:2508.13231), narrowing the gap between HBM and external memory “to within an order of magnitude.”

Two readings of this table matter for a purchase. First, the comparison variable for decode across these platforms is bandwidth tier and memory capacity, not FLOPS; the preprints themselves frame their evaluations that way. Second, the interconnect row is not trivia. Once weights or KV cache spill off the HBM into LPDDR5X or CXL-attached memory, the relevant bandwidth for those bytes drops from terabytes per second toward the 900 GB/s of NVLink-C2C (arXiv:2508.13231) or below, and the ceiling formula must use the bandwidth of wherever the bytes actually live.

Where the rule bends: batching, multi-token prediction, and prefill-heavy traffic

A rule you cannot bend is a rule you will misapply. Three conditions change the answer.

Large batches amortize weight reads. The roofline preprint hedges its own claim carefully: low arithmetic intensity holds “for most batch sizes and context length regimes” (arXiv:2508.10395). When many sequences share one weight streaming pass, bytes per token across the batch fall and arithmetic intensity rises. At sufficiently large batch, decode can migrate toward the compute roofline. If your serving stack runs high-concurrency traffic with aggressive batching, the bandwidth-first rule weakens; if you run interactive single-digit batch sizes for latency, it is at its strongest. The evidence does not quantify the crossover batch size, so treat the crossover as a measurement to make on your own stack, not a number to look up.

Speculative and multi-token decoding raise tokens per forward pass. The ceiling formula assumes one token per weight streaming pass. Multi-token prediction breaks that assumption. In a performance-engineering study on a single AWS EC2 instance with one NVIDIA A10G GPU, “MTP required only 221 contexts for 531 plain-text tokens, 224 contexts for 588 reasoning-intensive tokens, and 124 contexts for 321 tool-calling tokens” (arXiv:2609.35188), corresponding to “2.403, 2.625, and 2.589 completion tokens per context.” If each forward pass yields roughly 2.4 tokens instead of one, the effective bytes per token fall by the same factor and the bandwidth ceiling roughly doubles. The caveats cut both ways, though. The paper itself warns these tokens-per-context values are “not direct measurements of accepted speculative length,” and in that single-request setup the relaxed ceiling did not speed up generation: median per-token cadence was longer under multi-token prediction than under autoregressive decoding, 12.108 ms versus 10.363 ms for plain text (arXiv:2609.35188). This is one measurement on one GPU model, and acceptance behavior depends on workload. It shows the ceiling can relax, not that your latency will improve.

Prefill-heavy traffic flips the bottleneck. For long-prompt workloads, summarization over large documents, retrieval-heavy pipelines with big contexts, or embedding-style batch work, the compute-bound prefill phase dominates end-to-end time, and FLOPS genuinely is the binding constraint. The strongest evidence in the whole set is a study that took this asymmetry seriously enough to build hardware heterogeneity around it. In the MIST framework’s chat conversation use case, serving Qwen3-32B at 100 queries per second under a TTFT SLO of P99 ≤ 250 ms, the winning configuration put AMD MI350X accelerators on the compute-heavy prefill stage and NVIDIA H200-SXM on the bandwidth-sensitive decode stage: “This configuration produces 22% more tokens/$ at 17.4% lower per-hour deployment cost than a single-vendor configuration” (arXiv:2504.09775). The same study’s code-generation case chose a different pairing, Etched Sohu on prefill with TPUv6e on decode, delivering 49.3% more tokens/$ at 34.4% lower deployment cost than the best single-vendor option (arXiv:2504.09775). One co-design preprint, not a replicated benchmark, but it is the only result here that prices the decision, and it points the same direction the mechanism does: matching each phase to the resource it actually consumes beats buying one uniform fleet of whatever has the biggest FLOPS number.

Turning tokens per second into cost per token

The economic argument follows directly. If decode bandwidth sets your delivered tokens per second, then two instances with the same hourly price but different memory bandwidth deliver different tokens per dollar, and the higher-FLOPS instance can be the worse buy. The MIST result quantifies the upside of phase-aware placement at 22% more tokens per dollar in its chat use case, and 49.3% more in its code-generation case (arXiv:2504.09775), but you do not need heterogeneous hardware to use the principle. Even within one vendor’s lineup, the ratio of memory bandwidth to hourly price is the comparison that predicts decode economics, and peak FLOPS per dollar is close to noise for decode-heavy traffic.

Two cautions before you run this math. First, the MIST study publishes cloud rental prices per accelerator-hour, drawn from CoreWeave, TensorWave, and Google Cloud (arXiv:2504.09775): “H200-SXM $6.31, B200-SXM $8.60, GB300 $12.00, Etched Sohu $12.00, MI350X $7.50, MI355X $9.50, TPUv6e $3.00, and TPUv7 $8.00.” But no cited source pairs those prices with sustained bandwidth measurements, so a worked bandwidth-per-dollar table for specific rentals still needs your own quotes and your own measurements. Second, do not borrow provider API pricing as a proxy. A position paper on energy-per-token evaluation notes: “As of early 2026, posted prices across major providers still span over an order of magnitude on comparable per-million-token units” (arXiv:2605.11733). That spread describes API list prices, not self-hosted instance economics; it is evidence that token unit costs vary enormously across the market, not a number you can plug into a GPU rental decision. Your instance cost math needs your own quotes, your own measured derating of spec bandwidth, and your own phase mix.

How to size an inference node, and what this evidence cannot tell you

For interactive, decode-heavy serving, I would size in this order:

1. Capacity. Weights (at your quantization) plus KV cache at your p95 context length must fit in the fastest memory tier you can afford, or you are sizing an offload problem, not a GPU problem. The KV term is computable, not mysterious: roughly 2 × KV heads × head dimension × layers × bytes per element per cached token, scaled by batch and context, per the GenZ capacity model (arXiv:2406.01698).
2. Bandwidth. Compute the ceiling: active bytes per token (weights, adjusted for MoE activation if applicable; the GenZ analysis puts active parameters at 27% for Mixtral-8x7B and 15% for GPT4-1.8T (arXiv:2406.01698)), plus the KV cache term at your context length, divided by sustained bandwidth of the tier holding those bytes. Compare candidate GPUs and instances on bandwidth per dollar, not FLOPS per dollar.
3. Check the bending conditions. High sustained batch sizes, speculative or multi-token decoding, and prefill-heavy traffic each shift value back toward compute. If your traffic is prefill-dominated, invert the whole priority order, as the MIST phase split did with MI350X on prefill (arXiv:2504.09775).
4. Measure once before committing. The GenZ estimator suggests spec-based arithmetic lands within about 6% of real performance (arXiv:2406.01698), but a single benchmarked hour on your actual model and traffic costs little compared to a wrong fleet.

The honest limits: the supporting papers are arXiv preprint versions whose peer-review status was not verified; quoted bandwidths are vendor “up to” ceilings with no achievable-versus-spec measurements in the cited studies; and the crossover batch size where decode becomes compute-bound is asserted to exist but not measured. The bandwidth-first rule is well supported as a mechanism and as a direction. The specific numbers you need for a purchase order, sustained bandwidth on your model, tokens per dollar at your prices, and the batch-size crossover on your stack, are measurements only you can make on your workload. The framework tells you which measurements matter, and that, given how many GPU budgets get allocated by FLOPS-per-dollar, is most of the value.

Frequently Asked Questions

How does Mixture-of-Experts (MoE) sparsity affect the bytes moved per token?

For dense models, every parameter moves every token. The GenZ preprint quantifies the contrast: “For LLaMA2-7B, LLaMA3-70B, and GPT3-175B, all model parameters are utilized in each inference iteration, whereas in Mixtral-8x7B and GPT4-1.8T, only 27% and 15% of the total model parameters are activated per inference cycle” (arXiv:2406.01698). Applying dense-model arithmetic to a Mixture-of-Experts model overstates bytes per token by a factor of roughly four to seven, which makes a GPU look far too slow on paper for a model it would actually serve well.