How to Catch Broken LLM Quants in Model Registries Before Deploying

A registry tag is not a tested artifact. That is the finding that should change how you pull quantized models: a registry-scale execution census, Broken on Arrival (arXiv:2609.05881), ran 327 code-capable GGUF artifacts and reports that seven of 305 official Ollama library tags (a study-reported 2.3%) solved at most six of 164 HumanEval+ tasks. These files download, load, and generate tokens. They just do not work. Every confirmed failure was a low-bit quantization of a model with at most four billion parameters, and none of the registries the study examined documented functional testing of their uploads.

The practical consequence is that verification sits with you, the downloader. Catching these failures is cheap, and that is measured rather than assumed: the same study demonstrates a 15-task code screen that flags suspects, confirms them with a full HumanEval+ rerun, and ships as a runnable tool called quantcheck. What follows is a pre-deployment verification workflow grounded in that evidence, plus the cases where a screen is not enough and your fallback should be a pinned artifact or a local rebuild.

Why a registry tag is an unverified input

It helps to see model distribution the way the census paper frames it:

“Viewed as a data pipeline, model distribution moves a checkpoint through conversion, quantization and publication without a recorded functional check. For downstream users, a registry tag is therefore an unverified input.”

That framing matches what the study observed on registry pages: “None of the registries we examined documented functional testing of their files.” And the failures are not historical curiosities. Per the census, “The registry served five non-functional files unchanged, byte for byte, from our August 2026 census to October 2026,” with library pages that “gave users no failure warning and stated neither the quantization recipe nor whether a tag was tested.” Two months after measurement, the broken files were still live and unmarked.

Meanwhile, a GGUF selection guide advises choosing by bits, size, quality, and RAM (“For most users, Q4_K_M offers the best balance of quality, speed, and memory usage”), with no functional-verification step anywhere in its selection guidance. That page is one instance rather than a survey, but the general state of selection guidance is documented independently: a unified llama.cpp evaluation describes practical guidance on “which GGUF quantization to use” as “dominated by anecdotal advice and perplexity measurements, with no unified experimental framework for comparison.” That heuristic answers “which quant should I prefer” but assumes the file you download actually works. The census shows that assumption fails often enough to matter, and in a predictable tier.

What the census actually measured

Keep the scope precise, because it bounds what you can conclude. The study executed 327 code-capable artifacts: every eligible tag across 15 model lines in the official Ollama library (305 tags, 291 of them quantized), plus 22 popular Hugging Face community files, all tested with a code-generation screen. Within that scope, the study-reported non-functional rate was 2.3% of official tags, concentrated entirely in low-bit quantizations of models at or below 4B parameters.

Two further details matter for how much weight to put on the number. First, the seven-failure count is explicitly a lower bound: full HumanEval+ runs surfaced eight more files solving at most 16 of 164 tasks, seven of them in a “missing pre-tokenizer” group plus a gemma3-270M q8_0 build, all of which the study’s proposed gate would withhold from publication. Second, the gray zone is real. Twenty-six of the 305 eligible tags still lack a pre-tokenizer type two years after llama.cpp maintainers’ 2024 BPE pre-tokenization fix; those files tokenize prompts with a generic rule, altering 6 to 8% of prompt tokens (mostly indentation and spacing), yet 14 of the 26 still work, so the study declines to attribute that group. Brokenness here is not always binary, which is precisely why a screen tuned on known-healthy builds beats a hand-wave.

The pre-deployment checklist

The workflow below condenses the study’s detection approach into something you can run per artifact. The ordering follows the paper’s proposed publication gate: cheapest signals before full execution.

The quantcheck release implements the screen and scorer and bundles the study’s rebuild, calibration, and tensor-comparison scripts, so steps 3 through 5 do not have to be built from scratch. If you roll your own, the design principle from the paper is to tune pass thresholds on known-healthy builds of the same model line, because what counts as “passing” is model- and recipe-specific. In the study’s words, “Each model and recipe pair needs its own test. The same uncalibrated recipe works at 1.5B and fails at 3B, while nominally low-bit 0.5B files fall back to 4-bit types.” A threshold calibrated on one model family tells you little about another.

One design caution from a separate line of work: what you measure changes what you find. A PTQ study on diffusion transformers found that when prompts matched the training format, learned-preference judges stopped separating quantized variants, and the differences moved to axes the evaluation was not measuring. The lesson transfers: a spot-check built only from prompts in the model’s training format may rank variants on differences your judge cannot resolve. Include at least some out-of-format and reference-fidelity checks, especially if your production workload is chat rather than code.

Perplexity can flag catastrophic failures but cannot certify function

Perplexity is the cheapest quantitative check available, and the census measured it directly against its own failures. It did better than you might expect. On WikiText-2, the study reports, “each of the five non-functional library builds we measured has 27 to 5,700 times the perplexity of its fp16 model, while the 4-bit tags stay within about 5% of it,” and “Thresholds from 10 to 25 times fp16 would flag all five measured non-functional builds without executing generated code.” Cheap output statistics did worse: truncation and repetition rules flagged four of the seven failures plus six other tags, and missed Qwen q3_K_M, q3_K_L and llama3.2-1B q3_K_S entirely.

Read the caveats before adopting that threshold. The range was chosen after seeing these builds; “It needs an fp16 reference, and its false-positive rate is unknown because we did not measure the llama3.2-1B pair or the other tags.” Two of the study’s own calibrated Qwen Q3_K_S builds sit inside the flagged range at 18.6 and 18.8 times fp16. The deeper limit is stated just as plainly: “Perplexity is not a functional verdict, though. Our calibrated Qwen Q3_K_M build has about 6 times the fp16 perplexity yet solves 76 of 164 tasks, and perplexity cannot separate a corrupted file from an uncalibrated one.” The paper’s proposed publication gate therefore runs the perplexity pre-check first, where an fp16 build exists, and treats the 15-task screen as the acceptance test.

Even where perplexity looks normal, two independent 2026 studies show it can coexist with real damage. On Llama-3.1-8B-Instruct GGUF variants, a unified llama.cpp evaluation reports that several 5-bit schemes shared essentially the same perplexity (7.43 for Q5_0, Q5_1, and Q5_K_S, against 7.32 for the F16 baseline and 8.96 for Q3_K_S) while producing markedly different GSM8K and IFEval outcomes. The authors’ conclusion is worth reading directly: task sensitivity “depends on how quantization error interacts with the computation pattern demanded by a benchmark rather than on distributional fit alone.” Perplexity measures distributional fit on held-out text. It does not execute the behavior you are about to deploy.

The second result is stranger. In a feature-level audit, an INT7 quantization of Gemma-2-2B improved perplexity while degrading a study-reported 18.7% of active sparse-autoencoder features. A quant can look healthier than baseline on the aggregate metric and be damaged underneath it.

The census’s summary for practitioners draws the line: “The 15-task suite flags every failure reported here; a perplexity check caught the five we measured.” Perplexity is the cheap tripwire. Execution is the gate. Neither tells you why a file failed; in the paper’s words, “Neither attributes a cause, which still requires the rebuild comparison.”

Why provenance is not proof either

If size heuristics fail and perplexity fails, the next instinct is provenance: verify the recipe, maybe rebuild the file from the source checkpoint and compare. That instinct is also insufficient, and the census shows why in a satisfyingly uncomfortable way. The authors rebuilt five suspect files from the original weights, and the registry files matched their uncalibrated rebuilds at the tensor level. The files were faithful products of their recipes. They were also broken. As the paper puts it, “Execution and rebuilding answer different questions, because a file can match its recipe exactly and still not work.”

This is the strongest argument for gating on execution rather than metadata. Provenance checks still earn their place in the checklist: they catch corrupted uploads, mislabeled tags, and silent recipe drift, and a tensor comparison against your own rebuild tells you whether the file is what it claims to be. But a passing provenance check certifies identity, not function. Only running the model certifies function, and only for the behaviors you actually ran.

Fallback policy: pin, or rebuild locally

When an artifact fails the screen, or its provenance cannot be verified, you have two defensible moves.

Pin a known-good artifact. Once a specific tag has passed your screen, treat that artifact (by digest, not by mutable tag name) as the unit your pipeline references. The census’s persistence finding is the operational argument: five broken files sat byte-identical for two months, and nothing about registry pages would have told a new downloader. The same silence applies in reverse, though. A tag that works today can be re-uploaded tomorrow under the same name. Chasing the newest quant upload inherits all of the registry’s missing QA with none of the stability. Pinning converts an unverified input into a verified one, once.

Re-quantize locally from the source checkpoint. When no known-good artifact exists for the model and bit-width you need, building your own with a calibrated recipe is the controlled option. Calibration is not a formality at extreme bit-widths: in decoupleQ’s 2-bit experiments on Llama-7B (W2A16g64), accuracy increased monotonically with calibration dataset size, with 128 segments taking 4.3 hours against 19.5 hours at 2048 segments. The tradeoff is real compute time, so this path makes sense when the artifact will be deployed broadly or for a long-lived service, less so for an experiment you may discard. And per the census’s own observation that the same uncalibrated recipe can work at one size and fail at another, a locally rebuilt quant still goes through your screen before it ships. Rebuilding changes provenance, not the need to execute.

What the checklist cannot certify

Everything above certifies that a file runs and preserves screened behavior. It does not certify safety. Mind the Gap (arXiv:2505.23786) demonstrates that quantization itself is an attack surface: a model can be constructed to behave benignly in full precision and exhibit malicious behavior only after quantization, then get redistributed by community members who have no idea. The same work quantifies the exposure: over 70 thousand k-quant models on the Hugging Face Hub with more than 100 million downloads via popular libraries. A function screen will pass an adversarial quant, because the adversarial behavior is a function of the quantized model. If your threat model includes supply-chain manipulation rather than just sloppy uploads, execution screening is necessary and nowhere near sufficient; provenance, uploader trust, and full-precision behavioral comparison carry that weight.

Limits, and what to do on Monday

The prevalence numbers deserve their caveats in one place. The 2.3% figure comes from a single study of one official registry library: 305 Ollama tags across 15 model lines, plus 22 Hugging Face community files, all code-capable GGUF artifacts tested with a code-generation screen. The count is a lower bound, no AWQ or GPTQ artifacts were measured, and the census-wide prevalence has not been independently re-measured. The failures themselves do have outside confirmation: the paper reports that an outside reader reproduced the q3_K_M failure on a CPU-only Debian machine under a later Ollama release (0.33.3), recording identical output across runs and the registry’s manifest digest. Generalizing the rate to the broader Hub is inference, not measurement. What is not inference is the mechanism: no examined registry documented functional testing, broken files persisted unmarked for months, and the demonstrated failures cluster in a tier (low-bit quants of models at or below 4B parameters) that cost-constrained local serving selects for disproportionately.

The registry-level fix is obvious and the paper names it: “The first step is cheap: screen every low-bit tag at publication.” But your pipeline cannot wait for registries to adopt it. If you pull quantized artifacts for serving, I would put a perplexity pre-check where you have an fp16 reference and a generation-based task screen in front of every new artifact this week, gate low-bit ≤4B quants hardest, and freeze everything that passes behind a pinned digest. The screen costs a fraction of one deployment; the failures it catches were, as of this month, still being served.

Frequently Asked Questions

What is the quantcheck tool and what does it do?

The quantcheck release implements the screen and scorer and bundles the study’s rebuild, calibration, and tensor-comparison scripts, so steps 3 through 5 do not have to be built from scratch.

What should I do if a quantized model fails the verification screen?

Pin a known-good artifact. Once a specific tag has passed your screen, treat that artifact (by digest, not by mutable tag name) as the unit your pipeline references. The census’s persistence finding is the operational argument: five broken files sat byte-identical for two months, and nothing about registry pages would have told a new downloader.

Does a passing provenance check guarantee the model works?

But a passing provenance check certifies identity, not function. Only running the model certifies function, and only for the behaviors you actually ran.