When an LLM serving rig exhausts its VRAM, throughput does not degrade gracefully; it falls off a cliff. The reason is mechanical: every generated token requires reading the model’s working data, so decode speed is set by memory bandwidth, and the moment the working set spills off the card, those reads travel over PCIe to host RAM instead of staying on-package. Capacity planning therefore stops being a fit-or-no-fit question and becomes bandwidth accounting.
What actually happens when a GPU reads memory?
A single GPU load instruction traverses a hierarchy of storage that gets larger, slower, and farther from the compute units at every step: registers first, then L2 cache, then the on-board HBM or GDDR memory, and only then, if the data is not resident on the card, across the PCIe bus to the host’s system RAM. That hierarchy is the right frame for serving because the cost of a token is dominated by where its bytes happen to live at decode time.
The on-board tier matters because dedicated GPUs carry their own memory rather than sharing the machine’s main RAM, and Wikipedia’s GPU article notes that modern GPUs include hundreds or thousands of calculation units. Host RAM is not on the card: every byte that crosses to it goes over PCIe, a bus designed for peripherals, not for feeding thousands of cores a stream of weights. That split is what makes the hierarchy sharp rather than gradual.
Two properties of this hierarchy drive everything else in this article. First, the bandwidth gap between tiers is not a few percent. Moving the read path from on-board memory to host memory changes the economics of every single token, because the read now crosses a bus designed for peripherals rather than memory wired to the compute units. Second, the GPU does not choose which tier serves a read based on what is fastest for your workload. Data lands where capacity forces it to land. Once weights, KV cache, and activations exceed what fits on the card, the overflow lives in host RAM and its read path lengthens by an entire bus crossing.
Note what the cache tiers cannot do here. L2 exists to serve reuse, and decode offers almost none across the weight stream: every weight is touched once per token and the stream dwarfs cache capacity, so the effective bandwidth for the weight read is the bandwidth of whichever tier holds the bytes. Only a small hot set, normalization constants and attention biases among them, survives from token to token.
The operational consequence is that “memory” on a GPU is not one resource. It is at least three resources with different prices, and a serving stack that treats them as fungible will produce throughput numbers that look fine in a capacity spreadsheet and terrible in production.
Why does decode throughput track bandwidth instead of TFLOPS?
Autoregressive decode is a memory-bandwidth workload wearing a compute workload’s clothes. To emit one token, the model must read essentially all of its weights and the full KV cache for the active sequences, and it performs a comparatively small amount of arithmetic on those bytes. The ratio of floating-point operations to bytes moved sits far below the point where the compute units become the constraint, so tokens per second tracks how fast memory can deliver weights, not how fast the cores can multiply them.
The arithmetic makes the point without a benchmark. A matrix-vector multiply performs roughly one operation per parameter byte at batch size one, while the balance point of a datacenter GPU, the arithmetic intensity at which compute and bandwidth saturate together, sits far higher. The gap is structural rather than a tuning failure, and no kernel rewrites its way across it.
The spec sheet points you the wrong way. Wikipedia’s GPU overview frames card performance in terms of fabrication pathway size, clock frequency, on-chip cache sizes, and streaming-multiprocessor or compute-unit counts, and vendor marketing follows with teraflops. TFLOPS is the right metric for large-batch prefill and for training, where arithmetic intensity is high. For single-digit-batch decode, it is close to irrelevant. A card with half the TFLOPS and substantially more memory bandwidth will out-serve the bigger-compute part on decode-heavy traffic, and procurement decisions made on the headline number get this backwards.
The weight footprint side of the equation keeps growing. Current frontier models run to hundreds of billions of parameters, and even with aggressive quantization, a model of that class reads tens to hundreds of gigabytes from memory for every token at batch size one. Multiply that by the target tokens-per-second and you get a bandwidth requirement that only on-board memory can satisfy, which is why the working-set question in the next section dominates capacity planning.
The economics make this more than an academic point. The BenchLM leaderboard carries per-model pricing and latency alongside its quality scores, so cost per token is now a publicly compared number rather than a detail buried in an invoice. Cost per token at decode time is bandwidth per dollar. If your capacity model is built on TFLOPS and parameter counts, it is answering a question nobody is paying you to answer.
What happens when you run out of VRAM?
Running out of dedicated GPU memory changes the read path for everything that no longer fits: those reads cross PCIe to host RAM, and a bus designed for peripherals sets the pace for every token that touches host-resident data. How sharply throughput falls as oversubscription grows is a property of your hardware and serving stack, and it is the number to measure before it matters.
Serving stacks built around offload try to soften the crossing. Their usual design is to overlap transfer with compute, fetching the next layer’s weights over PCIe while the current layer calculates, hiding transfer latency behind arithmetic. What overlap cannot hide is a bandwidth shortfall: once the bytes a token needs exceed what the bus moves within that token’s time budget, overlap stops mattering and decode settles at bus speed instead of memory speed. Offload gives the cliff a slope. The slope still points down.
The trap is that the working set is not just the weights. It is weights plus KV cache plus activations, and the KV cache term grows with batch size and with context length. A deployment that “barely fits” at launch is one long-context user away from the cliff. Worse, the crossing is silent from the outside: the model still fits, requests still succeed, and the only symptom is that throughput collapsed and nobody’s capacity dashboard explains why, because the dashboard was tracking utilization and memory footprint rather than where the reads are being served from.
How do you build a bandwidth budget for KV cache, batch size, and sharding?
The worksheet is accounting, not sorcery: enumerate every byte the decode loop must read per token, identify which memory tier holds each class of bytes, and check that the tiers can sustain the resulting bandwidth at your target latency. Four steps cover most deployments.
Step 1: Weights. Parameter count times bytes per parameter, pulled from the model config and your quantization format, gives the per-token weight read. This is the floor. If weights alone approach the capacity of the card, everything below is an argument about how much headroom you do not have.
Step 2: KV cache. The standard formula is 2 (for keys and values) times layer count times KV head count times head dimension times bytes per element, per token, per sequence. Get the architectural constants from the model’s config.json, not from a blog post, because grouped-query and multi-head attention change the answer by large factors. Then multiply by your realistic context-length distribution and concurrent sequence count. This term is the one that grows at runtime and the one most capacity plans underweight.
Step 3: Place each class of bytes on a tier and price it. Weights and KV cache on HBM cost on-board bandwidth. Anything spilled to host costs PCIe bandwidth instead, at whatever efficiency your serving stack achieves over that bus. Compute the aggregate bytes per token per tier and divide each tier’s measured (not spec-sheet) bandwidth by that figure to get a ceiling on tokens per second per tier. The lowest ceiling wins. Treat the datasheet figure as an upper bound, not a promise, and measure realized bandwidth with a streaming benchmark on the actual card before trusting the arithmetic.
Step 4: Batch and shard deliberately. Batching amortizes the weight read across sequences, which is why decode throughput scales with batch size until you hit the compute-bound crossover or run out of KV cache space; the bandwidth budget tells you which limit arrives first. Tensor parallelism splits both weights and KV cache across cards, which converts an HBM-capacity problem into an interconnect-bandwidth problem: every layer now synchronizes over NVLink or PCIe, and the interconnect becomes another tier in the hierarchy with its own price. Shard because the budget says a single card cannot hold the working set, not because the GPUs were available in pairs.
What no leaderboard will tell you
The spine of this article is mechanism plus arithmetic, and both should be checked against measurement before you budget against them. The hierarchy of registers, cache, on-package memory, and PCIe is how the hardware is built, and the worksheet follows from it. But the numbers that decide your capacity, how far realized decode bandwidth sits below the datasheet figure and what throughput does as the working set crosses VRAM capacity, are properties of specific cards and serving stacks, not published constants. Treat them as measurements to take, not figures to quote.
The measurement gap in the public record is real but narrower than capability-only criticism suggests. The August 2026 BenchLM leaderboard tracks 399 models across 402 benchmarks1 and does surface serving economics: the board carries pricing and latency columns, though not every row is populated (Claude Mythos 5 lists $10.00/$50.00 per million tokens1 with its latency unmeasured), and its decision picks include ‘Best near-frontier value’ (Kimi K3, $15 output per 1M tokens1) and ‘Fastest measured’ (Ling 3.0 Flash, 397 tokens/sec1). What none of the 402 benchmarks1 measures is behavior under memory pressure: no score on the board tells you how a model’s throughput degrades once its working set exceeds VRAM. Capability and price are tracked in columns; the memory path is not tracked at all, and that is precisely the gap a bandwidth budget fills.
So how should you plan capacity?
Treat VRAM as a performance cliff rather than a fit-or-no-fit checkbox: a model that barely fits is one working-set increase away from severe degradation, so capacity planning should become bandwidth accounting across L2, HBM, PCIe, and host RAM. Size the KV cache, the batch, and any tensor-parallel shards against the memory path that serves every token, and treat any oversubscription as a measured degradation mode, not an acceptable fallback.
In practice that means three changes to how teams plan. Buy bandwidth, not FLOPS, for decode-heavy fleets, and let prefill and training workloads justify the compute-heavy SKUs separately. Hold headroom against the KV cache term, because context lengths drift upward over a deployment’s life and the cliff punishes optimism. And benchmark the oversubscribed regime on your own stack, because how throughput behaves past the fit boundary is exactly the number no published source can give you.
The limits of this framing deserve equal airtime. Bandwidth-first reasoning applies to autoregressive decode; large-batch prefill, training, and any future architecture that raises arithmetic intensity per token will shift the bottleneck back toward compute. And the transformer assumptions underneath all of this date to Google’s 2017 architecture, which the entire hardware stack now optimizes around; anything that changes the per-token memory read pattern changes the worksheet too. Until then, the memory path is the product you are actually selling, and the teams that budget it in bytes per second will keep their throughput while everyone else wonders where it went.
Frequently Asked Questions
Does the bandwidth-first planning model apply to large-batch prefill or training workloads?
No, the bandwidth bottleneck is specific to autoregressive decode where arithmetic intensity is low. For large-batch prefill and training, the workload shifts to compute-bound territory, making TFLOPS the primary constraint rather than memory bandwidth.
How does tensor parallelism alter the memory hierarchy constraints for LLM serving?
Tensor parallelism converts an HBM capacity problem into an interconnect bandwidth problem by splitting weights and KV cache across cards. This introduces NVLink or PCIe synchronization as a new tier in the hierarchy, adding a distinct price point for every layer’s data transfer.
What specific metric should operators measure to detect the onset of VRAM oversubscription?
Operators should measure realized tokens per second at working sets just below, at, and just past VRAM capacity. This empirical curve reveals the specific degradation slope for their hardware and serving stack, which is not captured by standard utilization dashboards.
Why do public leaderboards like BenchLM fail to predict serving cost for memory-heavy models?
Leaderboards track capability scores and API pricing but do not measure behavior under memory pressure. They lack benchmarks for throughput degradation when the working set exceeds VRAM, leaving the memory path economics untracked in public rankings.
