A 100 percent GPU utilization reading on an LLM serving node does not mean the GPU is doing useful compute work, and capacity plans built on that assumption will overspend. A new arXiv preprint, 2609.12923, observed in the feed on 2026-09-14, profiles exactly why on an H100 NVL. Its findings are author-reported and un-replicated, so treat every number below as a hypothesis about favorable conditions, not settled fact.
What the utilization counter actually counts
The instinct when a dashboard shows 100 percent GPU utilization is to conclude the hardware is saturated and the only way to serve more load is more hardware. The preprint’s central argument is subtler, and worth quoting directly because it inverts the usual complaint. The paper argues that “the problem is not that the counter is wrong, but that it collapses several different mechanisms into one number”.
That distinction matters for how you respond. If the counter were broken, the fix would be a better counter. If the counter is accurate but over-aggregated, the fix is decomposition: you need to know which mechanism is keeping the streaming multiprocessors occupied before you can say whether that occupancy translates into tokens. A GPU whose SMs are busy issuing memory requests they then wait on looks identical to a GPU whose tensor cores are chewing through dense matrix math, at least from the vantage point of the single percentage that nvidia-smi and default DCGM dashboards report.
This is the same aggregation trap that shows up elsewhere in capacity planning, where a single headline number hides the structure of the underlying demand. The preprint’s contribution, per its abstract, is to take that collapse apart on real serving stacks rather than synthetic microbenchmarks: the paper “profile[s] vLLM with FlashAttention-3 and cuBLASLt on an H100 NVL across cold prefill, warm prefill, and decode, sweeping sequence length and batch size”. That is a production-shaped workload, not a toy, which is what makes the mechanism claims worth a practitioner’s attention even before independent replication arrives.
Prefill vs decode: one utilization reading, two different workloads
LLM inference is not one workload. It is two phases with opposite bottlenecks, and the utilization counter cannot tell them apart.
Prefill processes the entire prompt in parallel. The matrix multiplications are large, the arithmetic intensity is high, and the tensor cores genuinely saturate. A high utilization reading during prefill usually does correspond to compute-bound work, because there is a full prompt’s worth of rows to multiply against the weight matrices.
Decode is different in kind. Each step generates one token per request, and each step reads the model’s weights from memory to do it. As Groundy has covered in the context of why throughput collapses when VRAM runs out, the cost of a decoded token is dominated by where its bytes live and how far they travel, not by arithmetic. The preprint puts the consequence plainly: the misrepresentation is “most severe during decode, where each request contributes only one new token and dense projection GEMMs become small-row matrix multiplications”.
So the same saturated reading on the same dashboard means, in prefill, “the tensor cores are full” and, in decode, “the SMs are busy waiting on memory while doing very little math per byte moved.” The counter is honestly reporting activity in both cases. It is the operator’s interpretation, not the measurement, that fails.
The practical reading rule follows directly: a utilization percentage is only interpretable alongside the phase mix of the serving workload at that moment. A request stream dominated by long prompts and short completions (summarization, document Q&A) spends its life in prefill, where the counter means roughly what people assume. A chatty workload with short prompts and long completions lives in decode, where the counter is at its most misleading.
The Hopper mechanism: 64-row fragments and wasted rows
The preprint’s most concrete mechanism claim concerns how matrix math actually executes on Hopper. On the bfloat16 GMMA path, the hardware executes operations in fixed 64-row matrix fragments. That is a hardware constant: the tensor core instruction consumes a 64-row tile whether or not 64 rows of real work exist.
During decode with a small batch, there are not 64 rows of real work. If eight requests are decoding concurrently, the projection GEMM has only eight rows of token data. The other 56 rows of each fragment are padding. The tensor core executes the instruction, the SM is occupied, the utilization counter ticks upward, and seven-eighths of the arithmetic performed serves no token.
This is the fragment fill problem, and it is the cleanest explanation the preprint offers for why nvidia-smi can read 100 percent while token throughput stays low: on the paper’s account, that combination is a coherent, expected observation rather than a paradox. The GPU is genuinely busy. It is busy multiplying zeros.
Batch size is the lever here, which is why the paper’s sweep over batch size and sequence length is the load-bearing part of the experimental design. Larger decode batches fill fragments more completely and amortize the weight reads across more tokens, pushing decode toward better efficiency per step. But batch size is constrained by KV cache memory and latency targets, so there is a real frontier: the batch that fills fragments may exceed the batch your latency SLO tolerates, and no utilization number will tell you where on that frontier you sit.
What to measure instead: eight counter-validated views
Rather than proposing a single replacement metric, the preprint decomposes utilization into eight counter-validated views, each derived from raw Nsight Compute reports and pinned to a named NCU counter or an explicit formula. According to the paper, these views map utilization gaps to concrete mechanisms: fragment fill, occupancy limits, stall signatures, wave quantization, and kernel selection, evaluated across four production models and six per-layer kernel roles.
The mechanisms themselves are worth understanding because each implies a different operational response:
- Fragment fill asks whether the tensor core tiles are full of real token rows. Low fill during decode points at batch size as the lever.
- Occupancy limits ask whether enough warps are resident to hide memory latency. Low occupancy with high utilization suggests the GPU is busy but serializing.
- Stall signatures identify what warps are waiting on: memory returns, barriers, or dependencies. A memory-stall-dominated signature during decode confirms the memory-bound hypothesis.
- Wave quantization captures the ragged tail when work does not divide evenly across SMs, leaving part of the chip idle in the final wave of a kernel.
- Kernel selection matters because the same operation can run through different kernels with different fill and wave behavior, which is why the profiling stack names cuBLASLt and FlashAttention-3 specifically. Groundy’s earlier coverage of RL-found CUDA kernels beating cuBLAS makes the adjacent point that kernel choice is a real, measurable variable in serving throughput, and that the free wins from autotuning should be captured before anything exotic.
| Signal | What it tells you | Where you get it | Cost to obtain |
|---|---|---|---|
| GPU utilization % | SMs had any active work | nvidia-smi, default DCGM | Free, already on |
| Per-phase SM activity | Which phase is busy, and how | DCGM per-counter views, profiler runs | Moderate instrumentation |
| Fragment fill / tensor core utilization | Whether tiles hold real rows or padding | Nsight Compute counters | High: profiler overhead, per-kernel runs |
| Stall signatures | What busy SMs are waiting on | Nsight Compute | High |
| Token throughput (per phase) | What users actually receive | Serving framework metrics (e.g., vLLM) | Low to moderate |
The honest summary of that table: the signals that discriminate real saturation from busy-waiting are also the ones that demand profiler-grade instrumentation. There is no free fix at the nvidia-smi level.
Capacity planning fallout
If utilization overstates useful work during decode, three common planning moves built on it become unreliable.
Rightsizing. Choosing GPU count or class from peak utilization readings assumes 100 percent means “no headroom.” If those peaks occur in decode-dominated windows, the true constraint is memory bandwidth and fragment fill, and a bigger or additional GPU of the same class may add less throughput than a batching or scheduling change. The conservative direction of the error is overspend: you buy capacity to relieve a compute saturation that is not compute saturation.
MIG partitioning. Splitting a GPU into MIG slices assumes the whole-device utilization signal decomposes cleanly across tenants. The preprint’s argument implies it does not: a slice showing high utilization may be running small-row GEMMs at poor fragment fill, in which case partitioning has not bought proportional throughput, only proportional counter activity.
Autoscaling triggers. A scaler that adds replicas when utilization crosses a threshold will scale out during memory-bound decode even when the bottleneck is not replica count but per-step memory traffic and batch geometry. Worse, scaling out reduces per-replica batch size, which worsens fragment fill, which keeps utilization high: a feedback loop that spends money to sustain the misleading signal. Triggers anchored to token throughput and per-phase SM activity break that loop, at the cost of the instrumentation described above.
This is the same pattern Groundy flagged when sizing agent memory as a capacity problem: measure the real unit of demand first, and let cheaper levers exhaust themselves before buying hardware.
How strong is the evidence?
Everything quantitative above comes from one preprint, one hardware configuration, and one serving stack. arXiv states that submissions are moderated but not peer reviewed, presented “as is” without warranty, and hosting conveys no approval of a work’s assumptions, methods, results, or conclusions. No independent replication of the fragment-fill or eight-view findings exists in the evidence available here.
That limits what you should do with the specifics. The 64-row GMMA fragment is an author-reported description of Hopper’s bfloat16 path; different hardware generations or kernels could shift where and how severely the counter misleads. FlashAttention-3 and cuBLASLt behavior on an H100 NVL may not transfer to other stacks. The four-model, six-kernel-role coverage is broad for a single study but narrow against the population of deployed serving configurations.
What survives independent of replication is the qualitative structure, because it aligns with mechanism-level facts about the architecture LLMs run on. The Transformer architecture these models use makes decode autoregressive and memory-bound by construction, a point Groundy’s coverage of unified-memory local inference arrives at from the hardware side. Serving cost is real enough that architectures like mixture-of-experts exist largely to cut per-input compute, which means the metrics driving capacity decisions carry direct dollar consequences. The preprint gives that known structure a concrete, counter-level explanation on current hardware. Pending replication, that is a strong hypothesis, not a settled law.
Practical verdict
Stop treating a 100 percent GPU utilization reading as evidence of compute saturation for LLM serving. During decode, a high reading can reflect memory-bound small-row GEMMs that keep the counter busy while a large fraction of each tensor core fragment processes padding. Re-anchor rightsizing, MIG partitioning, and autoscaling triggers to per-phase signals: SM activity split by prefill and decode, fragment fill and stall signatures where profiling budget allows, and token throughput as the user-visible ground truth. Accept that this instrumentation costs more than a default dashboard, because that cost is the price of a capacity signal that discriminates. And treat the preprint’s specific numbers as author-reported hypotheses about one favorable H100 NVL configuration until independent replication confirms them, because a plan is only as good as the evidence underneath its trigger thresholds.
Frequently Asked Questions
What specific hardware and software stack did the preprint profile?
Why does GPU utilization remain high during the decode phase?
the misrepresentation is “most severe during decode, where each request contributes only one new token and dense projection GEMMs become small-row matrix multiplications”.
What metrics should replace GPU utilization for capacity planning?
Re-anchor rightsizing, MIG partitioning, and autoscaling triggers to per-phase signals: SM activity split by prefill and decode, fragment fill and stall signatures where profiling budget allows, and token throughput as the user-visible ground truth.
