GPU Soft Errors in LLM Serving: How to Judge the Real Risk

Yes, GPU faults during LLM inference can corrupt your outputs without crashing anything, and the practical consequence is that crash-based monitoring alone cannot carry the correctness burden. The sharpest recent evidence for this is an instruction-level fault-injection preprint, “Analysis of LLM Vulnerability to GPU Soft Errors”, observed 2026-11-06, which is author-reported and unreplicated: every specific finding from it in this article carries that status. But its central premise does not rest on that paper alone. A cross-architecture injection study, a Rowhammer-on-ECC security paper, a fleet-scale corruption study from Facebook’s infrastructure, and an industrial telemetry study all point the same direction: silent corruption is a real production phenomenon, ECC does not close the gap, and the controls worth buying depend on which serving tier you are protecting.

The decision this article supports is concrete: whether to spend on ECC modes, memory-error monitoring, and output-level sanity checks, and how to triage that spend. The short version is that output validation belongs on high-stakes paths regardless of hardware, ECC is worth enabling where the hardware offers it but is not a correctness guarantee, and telemetry monitoring is a leading indicator rather than a safety net. The rest of this piece explains why, and where the evidence runs out.

Masked, SDC, DUE: a working vocabulary

The preprint’s most durable contribution is a taxonomy, one it explicitly credits to prior reliability work rather than claiming as its own. Quoting it directly:

“The consequences of LLMs under GPU soft errors can be categorized into three types [5, 6, 7]: (1) Masked, where faults are successfully masked and model inference remains unaffected; (2) SDC (Silent Data Corruption), where the model produces incorrect results without any system warning (in this work, we adopt the standard definition of SDC, where any token-level deviation from the fault-free output is classified as an SDC); and (3) DUE (Detected Unrecoverable Error), where unrecoverable errors cause program termination or crashes.”

Note how strict that SDC definition is: any token-level deviation from the fault-free output counts, even if the final answer still reads fine. That makes injection-campaign SDC rates an upper-bound sensitivity measure, not a user-visible-error measure. Keep this in mind whenever a paper reports an alarming SDC percentage; it is telling you how often the output stream diverges at all, not how often a user receives a wrong answer.

The operationally important split is SDC versus DUE. DUEs are the failures your existing stack already handles: CUDA errors, Xid events, process crashes, pod restarts. SDCs produce none of those signals. If your mental model of GPU reliability is “failures are loud,” the injection evidence says that model breaks, and it breaks unevenly across hardware.

The preprint also reports a second, more specific finding, quoted here with its unreplicated status attached:

“Insight #2: SDC and DUE exhibit distinct bit-level vulnerability patterns. SDC is dominated by high-order bits, whereas DUE is distributed more uniformly across bit positions, motivating different protection strategies.”

If replicated, that would matter for protection design: a detector targeting the exponent’s most significant bits, not the sign bit, would catch the corruption class that slips through silently, while crashes come from a flatter bit distribution. Treat it as a hypothesis with a plausible mechanism, not a design specification.

What injection proves, and what it can never say about your fleet

Instruction-level fault injection works by flipping a bit in a chosen register or instruction output during a model run, then observing the outcome. It is the right tool for answering “which operations are vulnerable?” and “what fraction of injected faults become visible corruption versus crashes?” It is the wrong tool for answering “how often does this happen in my datacenter?” Injection campaigns deliberately create faults at rates orders of magnitude above anything ambient; they establish vulnerability, not frequency.

No source in this evidence base measures ambient soft-error or SDC rates in a live LLM serving fleet. That gap has a direct consequence for how you should read the literature: every injection-derived SDC number is justification for controls, never a field error rate. A paper reporting that some fraction of injected faults corrupt outputs tells you the failure mode exists and is reachable; it cannot tell you whether to expect it weekly or once per fleet-lifetime.

There is also a live tooling constraint worth knowing before you try to reproduce any of this. The standard NVIDIA injection harness has bit-rotted against current drivers: its “precompiled core library, however, supports drivers only up to the 575 series” (syndrome-decoding paper on quantized GPU arithmetic), and its “kernel patching injection mode fails on our 580 series driver” (same paper). If your fleet runs that driver series, expect to patch or replace the injection toolchain before you can run your own campaign. That friction is itself a finding: the vulnerability-evidence base ages quickly even when the underlying hardware risk does not.

The field counterweight: corruption that repeats

The strongest reason to take silent corruption seriously is not any injection study. It is fleet telemetry from production. Facebook’s infrastructure team began investigating silent data corruptions in 2018 and spent the following three years analyzing detection strategies and their performance cost, per their paper on silent data corruption in a production fleet. Their central observation cuts against the comforting version of the “soft error” story: they report that silent data corruptions “are not limited to soft errors due to radiation and environmental effects with probabilistic models,” that they “can occur due to device characteristics,” and that “these failures are reproducible and not transient.”

That paper concerns a CPU-heavy production fleet rather than LLM GPU serving specifically, so do not read it as a GPU SDC rate. What it establishes is the failure’s character: some corrupted outputs come from specific, identifiable devices that corrupt deterministically and repeatedly. This changes the operational response. A transient radiation flip is a lottery; a reproducible device defect is a bad card quietly poisoning every request routed to it until someone notices. For LLM serving, where one device may hold a model replica for weeks, a reproducible corruptor is arguably the scarier case, and it is the case that output-level golden-set checks are good at catching: rerun a known prompt, compare the tokens, flag the replica.

You might hope telemetry would catch these devices instead. The evidence says it half-sees at best. An industrial GPU failure-prediction study describes the observability problem plainly:

“Temperature, power, utilization, clocks, and NVLink or PCIe traffic are indirect health indicators and are strongly confounded by workload. Optional counters may disappear for days, samples are not perfectly aligned, and healthy GPUs move among operating modes as jobs change.”

Two implications. First, indirect signals like temperature and utilization cannot distinguish “this GPU is corrupting outputs” from “this GPU is running a hot job,” so monitoring built on them produces leading indicators, not verdicts. Second, the counters themselves are an unreliable substrate: the optional ones may disappear for days, and the ECC and XID evidence that memory faults expose arrives in bursts tied to faults rather than as a continuously available stream. Any monitoring plan that assumes continuous counter availability will have blind windows measured in days. Memory-error counters are still worth collecting as a triage signal on best-effort tiers; they are not a substitute for checking outputs on the tiers where a wrong answer is expensive.

The ECC decision: what SECDED buys and what it does not

The naive position is “just enable ECC.” The adversarial evidence against it comes from Rowhammer research on ECC-protected GDDR GPUs: “Crucially, we discover that enabling ECC does not prevent privilege escalation exploits; we observe exploitable multi-bit errors with ECC enabled.”

The mechanism matters. SECDED (single error correct, double error detect) corrects one-bit flips and detects two-bit flips. Multi-bit errors beyond that can escape correction entirely, and single-bit flips are not the dominant corruption shape on GPUs: single-bit peaks are unit-specific and “never exceed 40% of non-special corruptions even when present” (GPU error-pattern study), “contrasting with CPU reports of 72–98% single-bit flips” (same study). If multi-bit corruptions are common, SECDED’s correction guarantee covers a shrinking share of the actual fault distribution.

What ECC does under stress is convert some would-be-silent faults into loud ones. The same Rowhammer study found that on an A6000 with ECC enabled, the attack triggered “detectable uncorrectable errors (DUE), i.e., double-bit errors, causing ECC failures and GPU crashes, at an average rate of 1 DUE per hour; on the more vulnerable A5000 GPU, we estimate it would suffer one DUE per 20 minutes on average.” Read those numbers carefully: they are attack-induced rates on GDDR6-based workstation GPUs (NVIDIA RTX A4000 through A6000), and the A5000 figure is an estimate, not an ambient failure frequency. Nobody’s fleet fails once an hour from background radiation. What the result shows is architectural: under multi-bit stress, SECDED on GDDR converts corruption into crashes rather than correctness. That is still a trade worth making, because a crash is detectable and a silent wrong token is not. But it means ECC spend on GDDR-class hardware buys failure-mode conversion, not immunity.

One scope limit before you generalize: these ECC findings are specific to SECDED-class protection on GDDR-based GPUs. They are not evidence about HBM ECC behavior on current datacenter accelerators, which no study cited here measures. The Rowhammer authors themselves note that “newer GPUs using HBM3/e and GDDR7 DRAM support on-die ECC” and leave exploits on those platforms to future work, so whether these results carry over to HBM parts is an open question.

Hardware axes: AMD, MIG, and the bit-position puzzle

Three hardware properties shift where your risk concentrates.

Architecture. The BitIR cross-architecture injection study reports the starkest architecture asymmetry among the studies cited here: “AMD. Faults are substantially more likely to produce corrupted outputs than explicit failures, making silent data corruption a primary concern. Mitigation on AMD platforms should therefore emphasize application-level output validation and error detection.” That is one study, not an industry consensus, but it is the only cross-architecture comparison in this evidence set, and its own authors draw the operational conclusion: on AMD platforms, output validation is the primary control, not a supplement.

Isolation topology. MIG partitions a single accelerator into up to seven isolated instances, each with dedicated compute resources and a private fraction of HBM. For multi-tenant serving, that partition is what a slice buys you: a tenant’s memory footprint is private, so cross-tenant memory interference is structurally limited. Two bounds apply. First, the isolation claim is documented for compute and HBM partitioning, not specifically for memory-error containment; whether a corrupting fault in one slice can affect neighbors is not settled by the studies cited here. Second, capacity constrains which models fit: a 70B-parameter model in BF16 needs roughly 140 GB for weights alone, so large-model replicas live on full accelerators or multi-GPU sharding regardless of MIG, and MIG-tier isolation is only available to the smaller models that fit the fraction.

Bit position and numeric format. Here the literature contains an apparent contradiction that resolves once you separate two measurements. The LLM preprint says SDC outcomes are dominated by high-order bit flips. The error-pattern study says flip frequency decreases from LSB to MSB within a corruption, with floating-point exponent bits (23–30 in FP32, 10–14 in FP16) least affected, which it attributes to simpler exponent logic than mantissa data paths. Both can be true: flips may be rarer in high-order positions, yet the flips that land there may be far more likely to survive to the output as silent corruption, because a flip of the exponent’s most significant bit produces a value change spanning several orders of magnitude that propagates through forward computation, while low-order mantissa flips often get washed out. The sign bit is the exception inside the high-order range: its flip only changes the sign and is often masked by subsequent nonlinear and normalization operations, so it contributes far less to SDCs than the exponent’s top bit. Frequency and vulnerability are different axes. For protection design, the preprint’s claim is the relevant one; for understanding raw corruption shape, the error-pattern study is.

The triage checklist

Putting the evidence together, the spend decision reduces to mapping each error source to its observable failure mode and the control that actually addresses it, tiered by what a wrong output costs.

Two checklist items are inference from the evidence rather than direct measurements, and I want to be honest about which. The golden-prompt replay recommendation follows from the Facebook finding that corruption is reproducible per device; the paper itself studied detection strategies broadly, not LLM golden sets. The AMD row rests on a single cross-architecture injection study. Both recommendations are cheap relative to the failure they address, which is why I would adopt them anyway, but they are engineering judgment built on the cited evidence, not measured baselines.

The tiering itself is also judgment. No study in this evidence base reports production SDC rates for LLM serving, so there is no measured threshold below which you can skip validation. The honest rule is asymmetric: the cost of a sampled output check is small and bounded, the cost of a silently wrong answer on a high-stakes path is set by your business, and the evidence establishes that the silent failure mode exists and is reachable on real hardware. That asymmetry, not a quantified risk budget, is what justifies the spend.

What would change this advice

Three pieces of missing evidence would sharpen every recommendation above. First, ambient SDC rates measured in a real LLM serving fleet: with those, tiering thresholds could be set from data instead of from asymmetry arguments. Second, replication of the anchor preprint’s bit-level findings, especially the high-order-bit SDC concentration, which would turn a plausible detector-design hint into a specification. Third, independent measurements of HBM ECC behavior on datacenter accelerators, to test whether the GDDR Rowhammer results generalize or are an artifact of the specific GDDR6 workstation cards tested.

Until that evidence arrives, the position to hold is narrow and defensible: GPU faults in LLM serving split into loud crashes and silent token corruption, the silent class is real and reproducible in production hardware, ECC converts some of it to crashes without guaranteeing correctness, and telemetry watches the problem through a dirty lens. Spend accordingly: validation on the paths where a wrong answer hurts, ECC where the hardware offers it, counters as an early-warning layer, and injection-study numbers filed under “motivation,” never under “rates.”

Frequently Asked Questions

How does the article define Silent Data Corruption (SDC) in the context of LLM inference?

(2) SDC (Silent Data Corruption), where the model produces incorrect results without any system warning (in this work, we adopt the standard definition of SDC, where any token-level deviation from the fault-free output is classified as an SDC)

Does enabling ECC on GDDR GPUs prevent all silent data corruption?

Crucially, we discover that enabling ECC does not prevent privilege escalation exploits; we observe exploitable multi-bit errors with ECC enabled.

What is the primary recommended control for AMD accelerator serving paths?

AMD. Faults are substantially more likely to produce corrupted outputs than explicit failures, making silent data corruption a primary concern. Mitigation on AMD platforms should therefore emphasize application-level output validation and error detection.