How to Estimate LLM Tokens per Second: Memory Bandwidth, Not FLOPS

If you want to know how many tokens per second a GPU will deliver on a local LLM chat workload, divide its effective memory bandwidth by the bytes of model data each token touches. That is the whole estimate. FLOPS, the number GPU buying guides lead with, barely enters into it for single-stream decode, because batch-1 autoregressive generation reads every weight once per token and performs almost no arithmetic on each read. The practical consequence: for chat on your own machine, rank GPUs by memory bandwidth and VRAM capacity, not TFLOPS, and when your measured speed falls short of the estimate, look at weight offload and framework overhead before blaming compute.

The rest of this article is that estimate made reproducible: where it comes from, a worked comparison for a roughly 32B model on a 24GB-class card, why real numbers land below the estimate, and the workloads where FLOPS do matter after all.

The 30-second estimate

The procedure has three steps:

1. Find the GPU’s effective memory bandwidth. Use a measured figure if you can; spec-sheet peak bandwidth overstates what kernels actually achieve.
2. Compute active bytes per token. For a dense model, that is the quantized weight size (parameters × bytes per weight), plus the KV-cache bytes read per step. For a mixture-of-experts model, count only the active parameters per token, not the total.
3. Divide. Tokens per second ≈ effective bandwidth ÷ active bytes per token.

This is a roofline: an upper bound, not a prediction. Real implementations undershoot it, sometimes substantially, and the section on diagnosis below covers where the gap goes.

Why FLOPS mislead at batch 1

The arithmetic-intensity argument is stated cleanly in the Whetstone engine’s README, a community-built 4-bit inference engine targeting this regime on consumer GPUs:

“At batch=1 autoregressive decode, every weight is read from memory once and used for a single multiply-add. That is an arithmetic intensity of ~2 FLOP/byte, against the ~120 FLOP/byte a Turing GPU needs to saturate its tensor cores. Decode is memory-bandwidth bound by roughly 60×.”

Two FLOPs per byte is a multiply-accumulate on each parameter, which is all decode does: one token in, one token out, every weight touched once. The tensor cores sit nearly idle. A card with double the FLOPS and the same memory bandwidth will produce roughly the same chat speed on the same model, while a card with double the bandwidth will roughly double it.

This is not only an LLM-community claim. A 2026 profiling study of query-based segmentation decoders on edge GPUs, an adjacent and non-LLM workload, reached the same conclusion independently, finding that “FLOPs are an unreliable proxy for latency: memory access cost and degree of parallelism can dominate runtime even when arithmetic complexity is low” (arXiv:2610.22377).

Worked comparison: a 32B-class model on a 24GB card

This section is worked arithmetic from the method above, not reported measurement. The numbers are ceilings derived from two inputs you can look up for your own hardware: effective bandwidth and bytes per token.

A 32B-parameter model costs roughly 64 GB of weight traffic per token in FP16 (2 bytes per weight), 16 GB at 4-bit quantization (0.5 bytes per weight), and 8 GB at 2-bit. Ignoring KV-cache traffic for a first pass, the decode ceiling is:

Two things about this table matter more than any single cell.

First, the 278 GB/s column uses a measured on-card bandwidth, not a spec sheet. It comes from the Whetstone reference machine, which also reports the 4-bit format it runs as 3.75× smaller than the source models (author-reported; treat as the repo’s own figures). What you should actually use is a bandwidth measurement on your card with your runtime, because achievable bandwidth depends on kernel quality and access patterns.

Second, the 4-bit row is why quantization dominates this decision. Going from FP16 to 4-bit roughly quadruples the ceiling: the repo’s own roofline for its 0.5B reference model reads 281 tok/s at fp16 against 1051 tok/s at int4, a 3.7× gap that tracks the byte ratio. It also shrinks the model from 64 GB to 16 GB, which is the difference between fitting in 24 GB of VRAM and not fitting. And fitting matters more than almost anything else, which brings us to the second column.

Why measured speed falls short of the estimate

If you run the estimate and then measure, you will almost always see a shortfall. The evidence points to three usual causes, in roughly the order you should check them.

Weight offload to host RAM. This is the catastrophic one. On the Whetstone reference machine, “Host RAM is ~46× slower than VRAM on this machine (6.0 GB/s over a PCIe 3.0 x8 link against 278 GB/s measured on the card)” (Whetstone README). That is the second column of the table above: if even a fraction of layers spill off the GPU, those layers run at single-digit-GB/s bandwidth, and since decode is serialized layer by layer, the slowest bytes set the pace. “It fits in 24 GB” claims that quietly offload layers to system memory are the most dramatic way this estimate and reality diverge. Budget for weights plus KV cache, which grows with context length and consumes the same VRAM.

Kernel-launch and scheduling overhead. Even with everything resident on-GPU, single-stream decode launches a long chain of small kernels per token, and the gaps between them are dead time. A 44-cell cross-GPU study of batch-1 decode quantified one recovery: “On H100 at ctx=2048, CUDA Graphs improves decode latency by 1.259× across N=10 fresh sessions, with a 95% bootstrap confidence interval of [1.253, 1.267]” (arXiv:2605.30571). That is a 1.26× speedup, roughly a 21% step-time reduction, from capturing and replaying the kernel sequence on a datacenter card, and direct evidence that a large share of the estimate-versus-measured gap there is scheduling overhead, not bandwidth. The effect is conditional, though. On L4, the lowest-bandwidth card in the same sweep and the one already running closest to its memory floor, the same intervention returned only 1.028×. And the overhead is largest in Python-dispatch-heavy stacks: the HuggingFace fp16 baseline on the Whetstone reference machine spends ~24 ms of every 27 ms token on Python dispatch, kernel launches and allocator churn. If your runtime offers CUDA Graphs or an equivalent graph-capture path, measure with and without it; on a fast card with a dispatch-heavy runtime it is the first cheap win, and on a card already near its bandwidth limit expect almost nothing.

KV-cache traffic. The estimate above counted only weights. At long contexts, reading the key-value cache for every prior token adds bytes per step, lowering the ceiling. The cited studies treat KV traffic as part of the memory-bound picture but do not give a consumer-card number for it, so I would measure at the context lengths you actually use rather than trusting a short-context benchmark.

A bounded hypothetical to make this concrete: suppose you compute a 17 tok/s ceiling for a 4-bit 32B model and measure 11. Before suspecting the GPU, check (1) whether any layers are offloaded, (2) whether graph capture is enabled, (3) whether your test context inflates KV reads. Each is a specific, checkable cause. This diagnostic order is inference from the measured findings above, not a tested recipe.

When compute does bind

The bandwidth rule has edges, and crossing them changes the hardware answer.

Prefill is compute-bound. The canonical split, from the POD-Attention paper: “The prefill phase is highly parallel and compute bound while the decode phase is memory bound.” Processing a long prompt is a big matrix multiplication, and FLOPS govern it. For interactive chat this usually matters less than it sounds, because profiling of OPT-2.7B found “the decode phase accounts for the vast majority of the total inference time for the OPT-2.7B model in all tested batch sizes. Although prefill becomes relatively more significant at large batch sizes, it remains below 5% even at the maximum batch size” (arXiv:2503.08311). But if your workload is document ingestion, retrieval over long prompts, or anything prefill-heavy, the compute-bound phase is what you wait on, and FLOPS comparisons regain their meaning.

Large batches change which bytes bind, not the rule. Batching amortizes weight reads across many tokens, so it is tempting to conclude that serving flips to compute. The Mind the Memory Gap study measured what actually happens at large batch: “the throughput plateau is caused by DRAM read saturation in the attention mechanism during the decode phase,” while “overall GPU resource utilization remains far from saturation at large batch sizes, especially in terms of compute capabilities.” Batching increases the arithmetic intensity of matrix-multiplication kernels, but “the arithmetic intensity of attention kernels remains nearly constant,” so attention’s KV reads saturate DRAM before the compute units fill. If you are serving many concurrent users, you are still buying bandwidth, but for a different traffic profile: attention and KV reads that grow with batch and context instead of amortizing like weights.

MoE models have their own rule: count active parameters and per-expert batch. For sparse mixture-of-experts models, what matters per token is the activated expert weights, not the total parameter count. The FORGE study of sparse-MoE inference makes the regime precise: “For DeepSeek-v3.2 (K=8, E=256), a serving batch of B=32 yields b_expert=1, i.e., each expert processes a single token, reducing its FFN projections to GEMVs. This pushes each expert’s FFN execution into the memory-bandwidth-bound regime.” In other words, even at a serving batch of 32, a wide MoE still runs its experts one token at a time, so the bandwidth rule holds much further into serving than a naive batch-size argument suggests. Compute only starts to bind at extreme scale: “the largest per-expert projections (DeepSeek-v3.2 and GLM-5 UpGate) reach 50% SM compute utilization only at b_expert ≈ 512” (arXiv:2610.16890).

Speculative decoding changes effective bytes per accepted token, since one weight pass can yield several tokens, and the Whetstone README quantifies the trade: “A 16-token pass costs 4.93 single-token passes with the weights in VRAM — so speculation can never beat 3.25× there however good the draft — but only 1.07 with them in host RAM.” Measured on Qwen2.5-3B, an 8-token draft ran at 0.97× on open-ended prose with the model resident, 1.56× on repetitive text, and 3.92× on repetitive text with 11 of 36 blocks offloaded to host RAM. Speculation is worth the most exactly where bytes are scarcest, and it can be a small net loss on resident open-ended text.

What to buy instead

For decode-heavy, single-stream local use, the ranking metric is effective memory bandwidth per dollar, subject to one hard constraint: the model, at your chosen quantization, plus its KV cache at your context lengths, must fit in VRAM. Within that constraint, a card with more bandwidth beats a card with more FLOPS for chat. The counterexample that clarifies the rule: a faster-FLOPS card with similar memory bandwidth changes almost nothing for this workload, because the tensor cores were never the bottleneck.

Two purchasing implications follow from the arithmetic. Quantization is the variable you control with the largest effect: 4-bit formats roughly quadruple the decode ceiling versus FP16 while also determining whether the model fits at all, which is exactly why engines like Whetstone are built around low-bit formats for this regime. And VRAM capacity is a binary gate, not a sliding scale: the moment weights spill to host RAM, you cross from the ~278 GB/s world to the ~6 GB/s world on the reference hardware above (Whetstone README), and no amount of FLOPS recovers that.

If your workload is prefill-heavy or expert-saturated MoE serving, invert the advice and shop on compute. Batch serving is not on that list: per the study above it stays a bandwidth purchase, just over attention and KV traffic instead of weight reads.

Verdict and the limits of the method

Estimate decode tokens per second as effective memory bandwidth divided by active bytes per token. Rank GPUs for local chat on bandwidth and VRAM capacity. Diagnose shortfalls in this order: host-RAM offload, kernel-launch overhead, KV-cache traffic. Revert to compute-first comparisons for prefill-heavy or per-expert-batch-≈512 MoE workloads; large-batch serving stays a bandwidth problem, with attention and KV traffic in the denominator instead of weights.

Hold the estimate as a ceiling, not a promise. The CUDA Graphs result shows real single-stream throughput sitting well below the bandwidth roofline for reasons that have nothing to do with bandwidth, and the worked table here is arithmetic from independently reported bandwidth figures (Whetstone README), not a measured benchmark. The adjacent-domain finding that FLOPs fail as a latency proxy supports the same direction, though it comes from segmentation decoders, not LLMs.

If you are about to spend money on the strength of this estimate, the honest procedure is: compute the ceiling for your model and quantization, confirm the whole thing fits in VRAM with room for KV cache, then verify against one measured tokens-per-second report for your exact model, runtime, and card. The estimate tells you which number to expect and, more usefully, which hardware property actually moves it.

Frequently Asked Questions

How do I estimate tokens per second for a local LLM chat workload?

If you want to know how many tokens per second a GPU will deliver on a local LLM chat workload, divide its effective memory bandwidth by the bytes of model data each token touches. That is the whole estimate.

Why does moving model weights to host RAM drastically reduce speed?

On the Whetstone reference machine, “Host RAM is ~46× slower than VRAM on this machine (6.0 GB/s over a PCIe 3.0 x8 link against 278 GB/s measured on the card)” (Whetstone README). That is the second column of the table above: if even a fraction of layers spill off the GPU, those layers run at single-digit-GB/s bandwidth, and since decode is serialized layer by layer, the slowest bytes set the pace.