Why a Faster GPU Won't Speed Up Local LLM Token Generation

An essay titled “It’s the memory latency, stupid” put a number on something local-LLM users already suspected: the author-reported arithmetic in the pivotlake.io piece holds that a modern CPU can execute roughly 4-8 instructions per nanosecond while a single RAM access costs 50-100ns, or about 400-800 instructions’ worth of idle time per memory fetch. The essay’s figures are one author’s back-of-envelope work, not a benchmark, but the mechanism it describes is the same one that governs token generation on your GPU. If you run llama.cpp, MLX, or ollama and you upgraded or rented a GPU by comparing TFLOPS, then watched tokens per second barely move, that is the expected outcome. Decode is a memory problem, not a compute problem, and the practical consequence is simple: read spec sheets for memory bandwidth in GB/s, keep the model’s weights on the fastest memory you have, and treat peak FLOPs as mostly irrelevant to how fast your chat replies stream.

The rule: every generated token re-reads the model

Autoregressive decoding works one token at a time. Each step has to touch essentially all of the model’s weights before it can emit the next token. A recent inference-acceleration paper states the consequence plainly: “every generation step requires loading the model weights and intermediate results from memory (e.g., High-Bandwidth Memory (HBM) for GPU servers), making throughput bottlenecked by memory bandwidth rather than compute” (Cats, arXiv:2605.11186). A platform-design study characterizes the same stage as one that “consumes (and generates) one token at a time,” relies on “a large cache of Key and Value projections of the input tokens,” and requires “high memory bandwidth (BW) and capacity” (arXiv:2406.01698).

That yields a rough ceiling you can compute in your head: tokens per second is bounded by memory bandwidth divided by the model’s size in bytes. A 70-billion-parameter model at 4-bit quantization occupies roughly 35 GB; on hardware that can stream weights at a given GB/s, you cannot generate more tokens per second than bandwidth divided by 35 GB. This formula is my inference from the memory-bound observations above, not a quoted measurement from the papers, and later sections cover why it overstates what you will actually see. But as an upper bound for single-user decode, it is the right first approximation, and it immediately explains the disappointing upgrade: doubling your GPU’s compute while leaving bandwidth unchanged doubles nothing.

Why a “faster” GPU stalls: batch-size-1 math and idle silicon

The reason compute sits idle is the shape of the arithmetic. A study of sparse-model inference describes the local-LLM regime exactly: “Short-context autoregressive decode, e.g., agentic tool-use chains, short-prompt chat, and single-user serving, is the regime where per-token latency is bottlenecked by the cost of moving weights out of HBM,” because each decode step produces one token and the feed-forward projections “collapse to low-arithmetic-intensity operations (GEMVs) whose wall-clock is set by weight transfer rather than arithmetic” (FORGE, arXiv:2610.16890). A GEMV is a matrix-vector multiply: one token’s activations against the whole weight matrix. There is almost no reuse of each weight, so the arithmetic units finish their tiny computation and then wait for the next slab of weights to arrive.

Hardware designers already build around this. One purpose-built inference processor pairs “a narrow GEMM engine, optimized for Grouped-Query Attention (GQA) delivering 39.3 TFLOPS (FP16), which is 8 times higher than its bandwidth” (HPU, arXiv:2504.16112). Even in that design, compute is provisioned as a modest multiple of the bandwidth budget, sized to keep the attention path fed rather than to maximize FLOPs. Silicon engineers are not spending transistor budget on compute the memory system cannot feed. Your purchase should follow the same logic.

Reading spec sheets for decode: GB/s and bits per weight

The two numbers that matter for single-user decode are accelerator memory bandwidth in GB/s and the model’s footprint in bytes (parameters times bits per weight, divided by eight). FLOPs, tensor-core counts, and clock speeds tell you almost nothing about how fast tokens stream in a chat session.

The papers cited here also show how to read a spec sheet. The HPU study’s profiling rig is an A100 PCIe offering “1.55TB/s of memory bandwidth, 40GB of memory capacity, and 312 TFLOPS (FP16, no sparsity) of computing performance” (HPU, arXiv:2504.16112), and the same paper’s comparison table lists the L40S at 864 GB/s and the H100 NVL at 3,900 GB/s. Two columns on one sheet, and for decode the GB/s column is the one that sets your ceiling. What those numbers cannot settle is the purchase most local users face: none of the cited work characterizes memory-bandwidth ceilings on consumer cards, or on AMD or Apple parts, and none reports llama.cpp, MLX, or ollama runs. One cited paper does evaluate inference throughput on a GeForce RTX 4090 at batch size 1, but only for its own speculative-decoding experiments (arXiv:2405.18628), not as a bandwidth profile, so I cannot tell you which specific card wins. Any article that quotes a tokens/sec figure for a named GPU without a benchmark citation should be treated skeptically. Before you buy, pull the memory bandwidth figure from the vendor’s spec sheet for the exact SKU you are considering, because bandwidth varies widely within a product family and marketing pages often lead with compute. The comparison you want is GB/s per dollar (or per rental hour), not TFLOPS per dollar.

There is also a reason to discount the number you find on the sheet. A profiling study of large-batch inference found that “DRAM read average throughput remains below 65% across all models during the decoding phase” (Mind the Memory Gap, arXiv:2503.08311). Real decode streams weights less efficiently than the spec bandwidth implies, so your achievable ceiling is lower than the naive bandwidth-divided-by-bytes estimate. Treat the formula as a comparison tool between candidates on the same runtime, not a prediction of absolute speed.

Where the rule bends

The bandwidth rule is strongest exactly where local users live: one user, short-to-moderate prompts, batch size one. It weakens or changes character in five named cases, and the boundaries matter because they change what you should buy. The first bend arrives before the first generated token: prefill, the stage that processes your entire prompt in a single forward pass, “has significant computing requirements” (arXiv:2406.01698), so time-to-first-token is the part of a request where compute still matters.

Two rows deserve prose. First, batching is usually cited as the escape hatch: serve many users at once, the arithmetic gets dense, and FLOPs start to matter. The measured counterpoint is striking: “Our findings challenge the prevailing assumption that large-batch inference transitions into a compute-bound regime and fully utilizes compute resources; instead, we demonstrate that DRAM bandwidth saturation remains the primary bottleneck, leaving significant compute resources underutilized” (Mind the Memory Gap). Even the regime designed to make compute matter stays bandwidth-limited on the systems profiled. That does not mean batching is useless (it raises total throughput per GPU), but it is not a reason to buy compute.

The compute crossover is real; it just sits far above single-user territory. The HPU paper frames the threshold as a ratio: the A100 it profiles has a performance-to-bandwidth ratio of “approximately 203,” so by standard roofline logic a kernel must reach roughly that operational intensity before compute, rather than memory, sets its pace (HPU, arXiv:2504.16112). FORGE’s H100 profiling locates the line for MoE expert projections: the largest ones reach 50% SM compute utilization (arXiv:2610.16890) only at a per-expert batch of about 512, which works out to a serving batch of roughly 16,384 tokens on DeepSeek-v3.2. Below batches like that, bandwidth is still the wall.

Second, long context is the one case where capacity can bite before bandwidth. The decode stage’s KV cache grows with sequence length and demands both bandwidth and capacity, “especially when processing long context queries” (arXiv:2406.01698). If your workload is summarizing long documents or agentic chains with big histories, memory capacity and KV-cache traffic join the weight stream as first-class constraints, and a card that fits the weights but not the cache will spill or fail.

Quantization is a bandwidth purchase, not an exemption

Fitting the ceiling formula to practice, quantization is the one lever that changes both numbers at once: it shrinks the model’s footprint in bytes, so each decode step moves fewer bytes, and it lets a larger model fit on faster memory. Through the ceiling formula, halving bits per weight, as in a move from 8-bit to 4-bit, halves the bytes per step and doubles the ceiling on a machine that is actually bandwidth-bound. That is the case for quantization as a speed lever, and it is arithmetic, not a measured comparison against other upgrades.

But quantization does not repeal the rule. The FORGE authors put it precisely: “Quantization thus reduces the bandwidth bottleneck and lowers the capacity ceiling, but we observe it eliminates neither” (arXiv:2610.16890). The accelerator still fetches every activated weight from memory on every step; the fetch is just smaller. What this evidence cannot tell you is the price: none of the studies cited here measures how much accuracy heavier quantization costs, so verify that tradeoff against per-model evals before you commit to an aggressive format.

Speedups with memory bills: speculative decoding and MoE

Two fashionable acceleration strategies interact with the bandwidth rule in ways marketing tends to omit.

Speculative decoding generates candidate tokens with a small draft model and verifies them against the large one, so it can produce several tokens per large-model pass. The catch is memory: an analysis of parallel prompt decoding found that “the extra runtime memory required for draft model storage and the KV cache—which scales linearly with sequence length—limits its practicality in edge devices, mobile environments, and long-context scenarios” (arXiv:2405.18628). On a memory-constrained local setup, the draft model and the extra cache compete with the weights you were trying to stream.

Mixture-of-experts models have the opposite marketing problem. An MoE model may activate only a fraction of its parameters per token, but the inactive experts still have to live somewhere. Across five frontier MoEs analyzed in FORGE, expert weights make up 94-98% of the parameter footprint and occupy HBM capacity continuously (arXiv:2610.16890). The “activated parameters” number on a model card is the compute story; the memory story is the full parameter count. And the two speedups compound badly: an edge-device study found that “in the decode stage, the incompatibility between MoE and speculative decoding increases expert activation,” making the expert memory traffic dominate (arXiv:2608.05303). That finding is from a single study on edge hardware, so I would treat it as a warning to measure rather than a settled law, but it points the same direction as the rest of the evidence: memory pressure, not arithmetic, is the recurring cost.

The PCIe wall

If the model does not fit in device memory, one alternative FORGE names is offloading weights to host memory, and its assessment is blunt: that path “severely bottlenecks decode performance to PCIe bandwidth limits” (arXiv:2610.16890). PCIe bandwidth is a small fraction of on-device HBM bandwidth, so a partially offloaded model generates tokens at closer-to-PCIe speed for its offloaded layers. This is also where Apple’s unified-memory machines enter the conversation: the whole memory pool is “device” memory, so a larger model can stay resident at memory-bandwidth speed. I cannot quantify that comparison; it is a structural argument, and you should verify the specific machine’s bandwidth before treating it as a recommendation.

The purchase rule follows: if you have to choose between a larger model spilling to host RAM and a smaller (or more heavily quantized) model that fits entirely in device memory, the resident model usually wins on decode speed. Check your runtime’s logs for offload before concluding anything about a GPU.

What I would do with this

For a single-user local setup, the decision sequence is short. First, pick the model and quantization that meets your quality bar, and compute its size in bytes. Second, find the memory bandwidth of each candidate GPU or machine from the vendor’s spec sheet, and divide: bandwidth over model size gives a relative ceiling you can compare across candidates. Third, confirm the model fits on-device at your target context length, KV cache included, because capacity failures and PCIe offload dominate every other term. Fourth, sanity-check against a real benchmark of your runtime on your hardware, since measured decode sits well below the nominal ceiling.

I would not pay a premium for compute in this regime, and I would treat “batching will use the FLOPs” and “quantization removes the bottleneck” as contradicted by the measured evidence above. The strongest caveat is the mirror image: the rule comes from mechanism and characterization in systems papers plus one author-reported blog essay, with no bandwidth profiles of consumer cards and no llama.cpp, MLX, or ollama runs. The rule “buy bandwidth, not FLOPs” is well supported; any specific card recommendation, including the one you are about to read in a comment thread, needs a spec sheet and a benchmark you can check.

Frequently Asked Questions

What two numbers should I check on a GPU spec sheet for local LLM decode speed?

The two numbers that matter for single-user decode are accelerator memory bandwidth in GB/s and the model’s footprint in bytes (parameters times bits per weight, divided by eight). FLOPs, tensor-core counts, and clock speeds tell you almost nothing about how fast tokens stream in a chat session.

Does quantization remove the memory bandwidth bottleneck for token generation?

But quantization does not repeal the rule. The FORGE authors put it precisely: “Quantization thus reduces the bandwidth bottleneck and lowers the capacity ceiling, but we observe it eliminates neither” (arXiv:2610.16890). The accelerator still fetches every activated weight from memory on every step; the fetch is just smaller.

What happens to decode speed if the model does not fit in device memory?

If the model does not fit in device memory, one alternative FORGE names is offloading weights to host memory, and its assessment is blunt: that path “severely bottlenecks decode performance to PCIe bandwidth limits” (arXiv:2610.16890). PCIe bandwidth is a small fraction of on-device HBM bandwidth, so a partially offloaded model generates tokens at closer-to-PCIe speed for its offloaded layers.