Does GPU Power Capping Actually Save Energy in LLM Inference?

On decode-heavy LLM serving, a GPU power cap mostly does not cut your energy bill. It enforces your per-node power ceiling, but in the memory-bound decode that dominates interactive traffic the cap never engages, so energy per token barely moves and neither does throughput; only where a large batch runs a long generation does the cap bite, and there the energy savings are real, paid for in throughput. That is the central finding of a recent phase-aware energy study, and it should change how platform engineers in colo facilities and power-budgeted datacenters think about nvidia-smi -pl.

One caveat belongs up front, because everything downstream depends on it: the headline numbers come from a single preprint with author-reported measurements, taken on one NVIDIA H200 SXM running vLLM in eager mode; the authors replicated their central findings on a second driver, but no independent reproduction appears in the record reviewed here. An independent power-trace dataset corroborates the mechanism, which raises confidence in the direction of the result, but every specific figure below is self-reported and should be re-verified on your own hardware before it drives a production power policy.

The assumption: cap the GPU, cut the bill

The reasoning most operators bring to this decision is intuitive. Power equals energy over time, so if the GPU draws fewer watts, it must consume fewer kilowatt-hours. Set the H200’s limit at, say, 400 W instead of 700 W and the meter should run slower.

That logic confuses two quantities. Instantaneous watts is a rate; what the utility bills is energy, the integral of that rate over time. If a cap slows the work down by exactly as much as it reduces the draw, total energy per request is unchanged. You have stretched the same kilowatt-hours across more wall-clock time. Whether a cap actually saves energy therefore depends on a question the intuition never asks: was the GPU near its power limit in the first place, and if the cap binds, does the workload slow down proportionally?

For LLM inference, the answer splits by phase, and the split is where the interesting engineering lives.

What the preprint actually measured

The core evidence is The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures, a 2026 preprint. Its scope is precise and narrow, in the authors’ own words: “We run on a single NVIDIA H200 SXM (HBM3e, 4.8 TB/s bandwidth, 989 TFLOPS BF16 dense peak, 700 W TDP) with NVIDIA driver 590.48.” The authors later repeated the core decode characterisation on a succeeding driver and report that both central findings reproduce, so the open gap is one GPU model, not one driver.

That scope matters as much as the results. The measurements were taken in vLLM eager mode without CUDA-graph capture, and the idle gaps between kernel launches are part of why decode draws so little power here. The preprint flags the execution paths it did not test: “Execution paths that remove the gaps, such as CUDA-graph capture or megakernels that run a whole forward pass as one persistent kernel, raise the draw toward the cap, change the clock response, and couple the components” (arXiv:2605.11999). The independent trace dataset’s vLLM runs, for instance, record “brief bursts from graph compilation and CUDA-graph capture” at startup (LLM-PowerTrace). Nothing here has been demonstrated on an A100, H100, or B200, or under SGLang or TensorRT-LLM. Treat what follows as a well-measured hypothesis about how decode behaves, not a settled property of GPU serving.

Why decode caps don’t bind

LLM serving alternates between two phases with very different hardware appetites. Prefill processes the whole prompt in parallel: it is compute-dense, keeps the streaming multiprocessors busy, and can push the GPU toward its power rating. Decode generates tokens one at a time. Each step reads the model weights and the KV cache from HBM, and the arithmetic per byte read is small. Decode is memory-bound: the compute units spend much of each step waiting on memory.

The preprint’s measurement of what that means for power is the article’s key number: “decode kernels wait on HBM and the GPU idles between kernel launches, so decode draws less than half of the GPU’s 700 W rating and throughput stays within a few percent under every cap we set” (arXiv:2605.11999).

Read that twice. Decode on this setup draws under roughly 350 W on a GPU rated for 700 W. A power cap is a ceiling. If the workload already lives far below the ceiling, the ceiling does nothing. The cap never engages, throughput is unaffected, and energy per token is unaffected, because nothing changed at all.

The authors state the consequence directly for the traffic patterns that dominate agentic and interactive serving: “decode-dominated serving (agentic multi-turn, streaming, latency-sensitive chat) holds the GPU in a low-power, memory-bound state where a 280 W cap delivers little energy reduction” (arXiv:2605.11999). Even that cap is the paper’s lowest, 280 W, which sits at 40% of the 700 W TDP, and it barely registers because the workload was already there (arXiv:2605.11999).

The operator’s version of this question has been turned into a pre-registered community experiment, though not yet into a result. tokenomics-bench is a harness designed to measure cost and energy per token under a p99 SLO, with vLLM batching, prefix caching, FP8, CUDA-graph modes, and power caps varied together on a different GPU generation. Its README is explicit about where it stands: “Status: harness built and pre-registered; hardware runs not yet performed. This README therefore contains no measured results.” Its motivating hypothesis matches the preprint’s mechanism: “On a decode-heavy workload a cap often never binds, because the GPU draws far below its limit, while locking the SM clock does change energy per token.” The repo poses that as the open question to test, crediting the preprint for the mechanism rather than claiming an independent observation. It is a harness worth watching, not corroboration.

What a cap actually buys: a ceiling whose price depends on the regime

If the cap rarely binds in decode, does it do anything? Yes, and the honest framing of its function matters for how you budget it.

A power cap is a safety envelope for the facility before it is an efficiency tool for the workload. Its job is to guarantee that no matter what the scheduler does, the node stays under its allotted ceiling. An independent time-resolved GPU power-trace dataset, LLM-PowerTrace, describes what happens at the boundary: “heavier AI load, whether more requests in flight, longer prompts, or larger models, raises power only until the GPU reaches its power limit; beyond it, additional work appears as longer request latency rather than higher power.”

The preprint delimits the same boundary for decode itself. The memory-bound regime, in its words, “ends where a large batch runs a long generation: each step streams a growing KV cache, decode turns sensitive to the SM clock, and the draw climbs past the cap.” There, per the preprint, the attention models approach the 700 W rating and “a 280 W cap cuts their throughput by up to 44%, while GDN keeps its throughput” (arXiv:2605.11999). For throughput serving at large batch and long generation, the cap does engage, and there it acts as a dynamic underclock: “the cap lowers the SM clock much as a lock does, and the best cap reaches energy comparable to the best locked clock.” Energy savings are available in that corner, but they are bought directly with throughput, and the 280 W setting is the pathological end of the trade.

So when load grows into the cap, the GPU does not get hotter; requests get slower. The economic consequence is worth stating plainly, and it splits by regime. If your datacenter contract charges for a power ceiling, the cap meets that obligation in both regimes. In the memory-bound regime that dominates decode-heavy traffic, the cap is inert: it neither reduces the kilowatt-hours you consume per token nor costs you latency, because it never engages. At large batch and long generation, where the cap does engage, energy per token does fall, to a level the preprint calls comparable to the best locked clock (arXiv:2605.11999), and you pay for that reduction in throughput. Capacity planning that assumes “we set a cap, so our serving energy dropped proportionally” is booking savings that, on this evidence, do not exist in the regime where most decode-heavy traffic actually runs.

This also relocates the real bottleneck. Once you accept that the power supply is no longer the constraint in decode, the levers that actually control cost and energy are scheduler behavior, batch size, and clock configuration. Those are software problems, and they are harder than setting a watt number.

Measure the right thing

The single most useful operational takeaway is about instrumentation. Instantaneous GPU power, the number nvidia-smi shows, is the wrong metric for this decision, for two reasons.

First, as above, a lower instantaneous draw can mean either “doing the same work more efficiently” or “doing the same work more slowly,” and only energy per request distinguishes them. Second, the prefill/decode split means a single power number per node averages two regimes with opposite relationships to the cap. A trace that shows prefill spikes pinned at the cap and decode valleys well under half the rating tells you the cap is doing real work for one phase and nothing for the other.

The practical measurement unit is energy per request, or per output token, decomposed by phase, under a stated SLO. The pre-registered tokenomics-bench harness is a reasonable template: vary the knob, measure energy per token at a fixed p99 latency target, and treat any configuration that misses the SLO as a different operating point rather than a saving (tokenomics-bench). “Saved 20% of energy at 2x p99 latency” is not an energy result; it is a different product.

The levers that do move energy per token

If the cap is a compliance device, what actually reduces decode energy? The evidence points to two mechanisms, plus one larger lever that has nothing to do with power management.

SM clock locking. The preprint’s alternative is to lock the streaming multiprocessor clock directly rather than cap board power: “Locking the SM clock instead lowers power without slowing memory-bound decode and recovers up to 32% of decode energy at minimal throughput loss” (arXiv:2605.11999). The mechanism follows from the memory-bound diagnosis: decode is waiting on HBM, so slowing the compute clock does not slow the workload, while a lower clock at lower voltage draws less power per second. The two levers act differently: a lock fixes the compute-core frequency while HBM stays at its rated clock, whereas a cap is a board-level ceiling that leaves clock selection to the driver and only lowers the clock once the draw actually exceeds it. Up to 32% is an “up to” figure from one preprint (arXiv:2605.11999), so plan around a smaller median until you measure your own models.

Phase-aware scheduling and consolidation. Festina, an energy-aware scheduling system for serverless LLM serving, attacks the problem at the scheduler rather than the GPU: “Comparison with four SOTA LLM serving systems and one DVFS-augmented system demonstrates that Festina reduces energy consumption by up to 56% while maintaining parity in SLO attainment” (Festina, arXiv:2606.30391v2). Two cautions apply. Festina operates in a serverless GPU-sharing setting, where consolidation opportunities are richer than on a dedicated colo node running one tenant’s traffic; do not bank the 56% figure for a dedicated deployment (arXiv:2606.30391v2). And its gains come from deciding which work shares which GPU when, which is a different intervention than anything nvidia-smi offers.

Quantization, for perspective. The LLM-PowerTrace dataset quantifies levers that shorten each decode step: FP8 and INT4 weights cut energy per output token by a median of about 30% and 39% respectively (LLM-PowerTrace), while an FP8 KV cache saves only around 3%, all measured under vLLM on A100, H100, and H200 with coefficients the authors call “specific to vLLM and the measured platforms”. Weight quantization is a bigger energy lever than any power-management knob discussed here, at the usual accuracy-evaluation cost. It belongs in the comparison because teams often reach for power policy before exhausting model-level options.

The clock-mismatch trap

The clock-locking result invites a lazy implementation: find the clock that saved 32% (arXiv:2605.11999), set it fleet-wide, move on. Festina’s own measurements say that will backfire. Forcing Qwen2.5 and Llama2 to run at the GPU clock that was optimal for Mistral increased their energy consumption by 10–16% (arXiv:2606.30391v2). The authors’ generalization is the sentence to keep: “an operating point that saves energy for one model or phase may waste energy for another.”

The reason is the same memory-bound logic run in reverse. The right SM clock for decode depends on the model’s arithmetic intensity per step: how much compute it does per byte of HBM traffic. That ratio varies with model size, attention architecture, batch size, and KV cache layout. A clock that is “low enough to be free” for one model can be below the knee for another, at which point you are slowing decode down and spending energy again. Any clock-locking policy needs a per-model, per-phase sweep, and it needs to be re-run when you change batch size or serving stack. This is real operational work, and it is worth acknowledging that “just lock the clocks” trades a one-line power cap for an ongoing tuning burden. For a small fleet serving one model, that trade is attractive. For a multi-tenant platform rotating models, the tuning cost may exceed the energy saved until the process is automated.

When the answer changes: prefill, attention architecture, and prefix caching

Everything above is about decode. Prefill is the phase where a cap can do real energy work, because prefill is compute-bound and can actually reach the power limit. But the preprint’s prefill findings carry their own measurement caveat: they were taken with prefix caching disabled.

Under that condition, the paper reports an architectural split. Recurrent architectures with linear-time mixing avoid a prefill penalty on long prompts, undercutting quadratic attention and lowering total request energy from the first output token at large batch sizes (arXiv:2605.11999). With prefix caching enabled, vLLM’s default configuration, repeated prompt prefixes never re-enter prefill at all, which shrinks prefill’s share of total energy and further weakens the case that a cap saves anything system-wide.

For an operator, the actionable question is your own phase mix. A workload dominated by long, unique prompts (document analysis without shared prefixes) has a substantial compute-bound prefill component where power management can bite. A workload dominated by chat and agentic loops with heavy prefix reuse is almost entirely decode, and the cap is decorative from an energy standpoint. You cannot answer this from the preprint; you answer it from your own request logs.

Verdict

Keep the power cap if your facility contract or rack budget demands a hard per-node ceiling. That is what it is for, and the LLM-PowerTrace data confirms it works as a ceiling. But stop booking it as an energy saving on decode-heavy traffic. On the evidence available, decode on an H200 draws under half the 700 W rating, a 280 W cap barely moves energy per token, and the compliance cost shows up as latency when the cap does engage (arXiv:2605.11999, arXiv:2610.38315v1).

If the goal is actually fewer kilowatt-hours per token, the supported path is: measure energy per request split by phase under your real SLO; trial SM clock locking per model, expecting something under the reported 32% best case (arXiv:2605.11999) and re-tuning per model given the 10–16% mismatch penalty (arXiv:2606.30391v2); evaluate phase-aware scheduling if you run shared or serverless capacity; and consider weight quantization, which the trace data suggests is a larger lever than any of the power knobs.

Before changing production policy, replicate on your own stack. The gaps in this evidence are specific: one GPU model, with the central findings self-replicated by the authors on a second driver but not independently reproduced, eager-mode vLLM without CUDA graphs, a single preprint behind the headline claim, and no NVIDIA power-management guidance in the picture at all. If your serving stack captures CUDA graphs, the idle-between-launches behavior that makes decode so cheap here may look different, in either direction. I would treat the preprint’s mechanism as a strong prior and its numbers as a starting hypothesis, and run the tokenomics-bench-style sweep, caps and clocks varied together against energy per token at your p99 target, on a single node before touching the fleet. The experiment is cheap. The wrong fleet-wide power policy is not.

Frequently Asked Questions

Does a GPU power cap reduce energy consumption during LLM decode?

decode kernels wait on HBM and the GPU idles between kernel launches, so decode draws less than half of the GPU’s 700 W rating and throughput stays within a few percent under every cap we set

What is a more effective method to reduce energy in memory-bound LLM decode?

Locking the SM clock instead lowers power without slowing memory-bound decode and recovers up to 32% of decode energy at minimal throughput loss