Why the Same LLM Prompt Returns Different Outputs Across GPUs

Same prompt, same model, same seed: different tokens. If your serving stack does this across GPUs, runs, or batch sizes, the cause is not a bug in your eval harness. It is floating-point arithmetic behaving exactly as specified, combined with kernel and batching choices your inference engine makes on the fly. A September 2026 preprint demonstrates both the mechanism and a fix, and the fix changes what you should promise about reproducibility.

The symptom, and why your harness is not broken

Engineers running vLLM, SGLang, or TensorRT-LLM keep hitting the same wall: a prompt that produced one completion on an A100 produces a slightly different one on an H100, or a different one again when the server is under load versus idle. Golden-output tests flake. Eval scores drift by fractions that compound across a benchmark suite. Audit commitments about reproducible outputs become quietly unenforceable.

The first thing to rule out is the usual suspect. Random seeds do not fix this. Seeds govern sampling randomness; they say nothing about how floating-point sums get computed. A new arXiv preprint on cross-GPU determinism states the root cause plainly: “floating-point non-associativity combined with hardware-dependent kernel selection.” Both are deterministic-looking sources of divergence that no seed touches.

Failure mode 1: kernel selection and the arithmetic itself

Floating-point addition is not associative. Computing (a + b) + c can differ in the last bits from a + (b + c), because each intermediate result rounds. In a single kernel with a fixed execution order, that rounding is at least repeatable. The trouble starts when the order changes.

GPU kernels choose tiling, split strategies, and reduction orders based on the hardware they run on: SM count, cache sizes, available instructions. An H100 and an A100 running the “same” matrix multiply will typically run different kernel configurations, which sum the same numbers in different orders, which produces different low-order bits. Those bits feed the next layer, then the next, and eventually a logit ranking flips and a different token wins the argmax. One flipped token early in generation diverges the whole completion.

The preprint demonstrates this is real by inversion: its headline result is that its own kernels achieve linear-layer outputs that are “bitwise identical across NVIDIA Ampere, Ada, and Hopper GPUs.” You only need to prove bitwise identity across three GPU generations if bitwise divergence across those generations is the default. It is.

Failure mode 2: your batch size is part of the computation

The subtler failure mode survives even perfectly deterministic kernels. The preprint cites He and Thinking Machines Lab (2025), who “attribute serving nondeterminism to the lack of batch invariance: varying batch sizes change GEMM tilings and reduction orders, producing different outputs despite deterministic kernels.”

The mechanism is the same floating-point story one level up. When a GEMM processes a batch of 8 requests versus 64, the library picks different tilings to use the hardware well, and different tilings mean different reduction orders for what is mathematically the same row of your request. Your output depends on who else happened to be in the batch with you.

This is where production serving stacks walk straight into the trap. vLLM’s own documentation lists its core features as “continuous batching of incoming requests, chunked prefill, prefix caching.” Continuous batching is the throughput win that makes vLLM worth running: requests join and leave the batch every decode step. It also means batch composition, and therefore the effective GEMM shapes your request passes through, changes moment to moment with load. Chunked prefill and prefix caching reshape workloads the same way. Prior Groundy analysis of the launch-tax economics of batching explains why engines batch this aggressively; the reproducibility cost is the unpriced side of that trade.

Failure mode 3: the exposed surface in real stacks

Beyond batching, serving stacks expose exactly the kernel-selection surface the preprint implicates. vLLM lets operators select among attention backends including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton. Different kernels, different reduction orders, different bits. Attention kernels that use atomic accumulation for parallel reductions add another nondeterministic element, since atomics complete in scheduling-dependent order.

TensorRT widens the surface further. NVIDIA’s TensorRT documentation describes compilation to engines “with support for mixed precision (FP32/FP16/BF16/FP8/INT8/FP4/INT4), dynamic shapes, and specialized optimizations for transformers and large language models.” Every precision format rounds differently. Every dynamic shape can trigger a different kernel path. An engine rebuilt with a different TensorRT version, a different CUDA toolkit, or a different target GPU is a new arithmetic, not the same one recompiled.

So the failure-mode map for a deployed stack has three layers, each independently capable of flipping tokens:

Fixing one layer leaves the other two live. That is why “we set the seed” and “we enabled the deterministic flag” both keep failing audits.

What pinning determinism actually costs, and a surprising result

The preprint’s mitigation targets the first two layers directly. It uses fixed-configuration fused-upcast GEMM kernels that “load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its SM count, or kernel scheduling.” For small-batch decode shapes, it uses a split-K scheme whose partial sums combine in a fixed ascending order with no atomics, removing the scheduling-dependent accumulation entirely.

The conventional assumption is that determinism taxes throughput: fixed configurations forgo the per-shape tuning that makes fast kernels fast. The paper reports the opposite for its implementation: the deterministic kernels run “1.17 to 3.1× faster end-to-end than the state-of-the-art solution and cutting weight-memory traffic in half.” If that holds, the argument that reproducibility must cost performance weakens considerably, at least for linear layers.

Two caveats matter before you budget around that number. First, the result is author-reported in a single preprint with no independent replication. Second, the speedup compares against a prior deterministic state of the art, not against the fastest nondeterministic path your serving engine currently runs. The throughput question that matters operationally, “what do I lose relative to stock vLLM on my hardware,” is not answered by this evidence.

Batch invariance has a boundary

The third layer is only partially addressed. The preprint’s kernels are batch-invariant “within its decode bucket: a given request’s outputs are bitwise independent of how many other requests share its batch.” The qualifier is doing real work. Requests grouped into the same decode bucket are protected; the guarantee does not obviously extend across prefill chunking, bucket boundaries, or changing prefix-cache state.

And the scope of the whole result is linear layers. Attention, sampling, and the surrounding serving machinery are outside the demonstrated bitwise-identity claim. No deployed vLLM, SGLang, or TensorRT-LLM configuration has been shown to achieve end-to-end bitwise determinism. The preprint is best read as proof that the hardest kernel-level problem is solvable, not as a feature you can enable today.

The revalidation tax nobody budgets for

Suppose you do pin outputs: fixed engine version, fixed kernels, fixed hardware, fixed precision. Your golden tests now pass. The cost arrives on a schedule you do not control.

Every engine release invalidates your pinned baseline. vLLM shipped stable v0.30.0 on September 22, 2026, and the project releases frequently; its creators have also launched a commercial startup, Inferact, which suggests the development pace will not slow. Each upgrade is a choice between a stale engine and a full re-baselining of every golden output.

Hardware moves are worse. TensorRT 11.3.0 is built against CUDA Toolkit 13.4 and drops DLA support, and per NVIDIA’s documentation, Jetson deployments must stay on TensorRT 10.x. A fleet spanning datacenter GPUs and Jetson edge devices cannot even run one engine version, let alone one arithmetic. Any reproducibility promise that crosses that boundary is unkeepable, and any cluster migration to a new GPU generation reopens the kernel-selection problem unless your deterministic kernels genuinely cover it.

The second-order effect compounds: the more you invest in pinned-output testing, the more expensive every model update, engine upgrade, and hardware refresh becomes, because each one triggers revalidation. Determinism work shifts engineering effort from throughput tuning to reproducibility maintenance, permanently.

Pin or accept: a decision guide

The evidence supports a split strategy, not a universal one.

Accept nondeterminism (the default). For eval harnesses, regression suites, and CI golden tests, stop asserting exact output equality. Assert properties of distributions: pass rates over sampled runs, score bands, similarity thresholds, structural invariants. This matches how the underlying system actually behaves under continuous batching, and it survives engine upgrades and hardware moves without re-baselining.

Pin determinism (scoped and deliberate). Reserve bitwise pinning for audit-critical paths where an exact replay is a contractual or regulatory requirement. If you take this path, the evidence dictates the checklist:

• Fix the decode bucket and execution configuration, not just the seed. Batch invariance within a bucket is the strongest guarantee the current literature demonstrates.
• Pin the full arithmetic stack: engine version, CUDA version, kernel backends, precision format, GPU generation. TensorRT’s CUDA 13.4 dependency and Jetson split show how fast these drift.
• Budget revalidation as a recurring cost on every release, migration, or precision change, and decide in advance who pays it.
• Treat any vendor “deterministic mode” claim with the paper’s taxonomy: does it cover batch invariance, cross-GPU scope, and attention, or only same-device reruns of linear layers?

What the evidence does not show

Every quantitative claim here traces to one author-reported preprint. The 1.17, 3.1× speedup, the halved weight-memory traffic, and the Ampere/Ada/Hopper bitwise identity all cover the paper’s own fused-upcast GEMM kernels on linear layers. Nothing has been independently replicated, nothing demonstrates end-to-end determinism in vLLM, SGLang, or TensorRT-LLM as deployed, and the mitigation costs for real serving configurations are unmeasured. The vLLM v0.30.0 release date and the Inferact funding detail rest on Wikipedia rather than on primary announcements.

The honest summary: the root causes of cross-GPU divergence are now well identified, kernel-level fixes demonstrably exist, and the performance penalty for determinism may be smaller than assumed. But no shipped serving stack closes the gap today. Until one does, distribution-level assertions are the robust default, and bitwise reproducibility is an engineering investment you scope, fund, and re-validate, not a flag you set.

Frequently Asked Questions

Does setting a random seed fix cross-GPU output differences?

Random seeds do not fix this. Seeds govern sampling randomness; they say nothing about how floating-point sums get computed.

How does batch size affect LLM output determinism?

varying batch sizes change GEMM tilings and reduction orders, producing different outputs despite deterministic kernels.

What is the performance cost of deterministic kernels?

the deterministic kernels run “1.17 to 3.1× faster end-to-end than the state-of-the-art solution and cutting weight-memory traffic in half.”