4-Bit vs 8-Bit Quants: Why Accuracy Benchmarks Miss Distribution Drift

A single-author, unreplicated preprint (arXiv:2609.07664, v2 posted 2026-09-09, author-reported) argues that zero-shot accuracy is the wrong yardstick for quantized models: it reads only the argmax token, so quants at different bit-widths can match the base model’s benchmark score while their token-level output distributions drift apart. If that argument holds, accuracy parity stops being sufficient evidence for a production quant swap, and the evaluation burden shifts to distribution-level comparison before rollout.

What argmax accuracy cannot see

The mechanism is simple enough to state in one sentence, which is part of why it has gone unnoticed for so long. Zero-shot benchmark scores are computed from the single highest-probability token at each decision point. Quantization perturbs weights, which perturbs logits, which reshapes the probability distribution across the whole vocabulary. If the top token survives that perturbation, the score is unchanged, no matter how much the rest of the distribution has moved.

The preprint, “Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs” by Shahzeb Qamar, states the problem directly: evaluating quantized models is “largely based on zero-shot task accuracy, which depends solely on argmax predictions and is insensitive to changes in the underlying predictive distribution.” A model that assigns high probability to the correct token and one that barely edges out a near-tie runner-up can post identical accuracy while behaving very differently the moment anything other than the top prediction matters.

The second half of the critique is more corrosive for rollout decisions. Under progressive quantization, the paper reports, accuracy “can exhibit unstable, non-monotonic behavior,” which “mask[s] substantial fidelity loss relative to the BFloat16 (BF16) uncompressed base model and provid[es] misleading deployment signals.” Picture a team stepping down through quantization levels, watching benchmark scores wobble within noise, and concluding that Q4 or even Q3 is safe. The score they watched stayed flat. The quantity they never measured, the distance between the quant’s output distribution and the base model’s, kept growing.

Not from the paper. The downstream failure modes that motivated this article (inconsistent RAG answers, flaky agent tool-call formatting, flips on long-tail prompts) are extrapolations from the mechanism above. None appear as measured results in the paper’s abstract. They are plausible because they are exactly the workloads that consume more than the argmax token, but treat them as hypothesis, not finding.

The divergence toolkit: JSD and TV against the BF16 base

The paper’s proposed fix is to measure the thing accuracy ignores. It computes “statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between outputs of full-precision and quantized models,” specifically between full-vocabulary predictive distributions at the token decision boundary.

Both metrics are standard and cheap to reason about. Total Variation Distance asks, for the same next-token decision, how far apart two models’ probability assignments can get on any single token. Jensen-Shannon Divergence is a smoothed, symmetric measure of how distinguishable two distributions are. Neither requires ground-truth labels, which is the operational attraction: you need only run the BF16 base and the candidate quant over the same prompts and compare what they emit.

The headline result is directional rather than granular: “divergence metrics generally increase under stronger quantization, complementing task accuracy with a fidelity signal.” Two things are worth noticing in that sentence. The signal moves in the direction you would expect, which is the sanity check any fidelity metric must pass. And the authors position it as a complement to accuracy, not a replacement. Divergence tells you how far the quant sits from the base model; it does not tell you the base model was worth imitating.

What the 120 runs covered, and what they didn’t

The evidence base is a 120-run experimental matrix: five foundation architectures, four reasoning benchmarks, and progressive quantization regimes from uncompressed BF16 down to Q2_K.

Here the honest limitation matters for anyone quoting this work. The paper’s abstract does not print an explicit 4-bit-versus-8-bit parity pair. The regimes reported run from BF16 to Q2_K, and the specific scenario in this article’s title (a 4-bit and an 8-bit quant both matching base accuracy while drifting differently) is a framing of the paper’s mechanism plus its monotonic trend, not a printed result. By the paper’s own “generally increase” finding, an 8-bit weight quant should sit closer to base than a 4-bit one, but per-architecture numbers, explicit 8-bit comparisons, and any RAG or agent-workload evidence are all absent from the material reviewed for this piece. The framework is the durable contribution; the specific numbers await reading the full paper and, more importantly, independent replication.

”8-bit” is not one thing

Before arguing about 4-bit versus 8-bit, name the scheme, because in the llama.cpp ecosystem those labels hide real variation. Per Hugging Face’s GGUF documentation, Q8_K is an 8-bit type with 256-weight blocks that is “Only used for quantizing intermediate results,” with all 2-to-6-bit dot products implemented against it. It is not a weight quantization type. When someone says “the 8-bit quant,” they almost always mean legacy Q8_0 weights.

The 4-bit side is just as split: legacy Q4_0 (round-to-nearest, 32-weight blocks) versus K-quant Q4_K versus importance-matrix IQ4_XS are three different schemes at roughly similar memory footprints. Wikipedia’s GGUF article groups Q8_0 with Q4_0 as “simpler legacy quantizations” that the block-based K-quant variants “generally” outperform at a given bit-width. So the practical question is never “4-bit or 8-bit.” It is “which scheme, at what bits-per-weight, against what reference model.”

At 4 bits, the scheme matters as much as the number

This is where the preprint delivers its most immediately actionable finding: “Across tested llama-cpp schemes, mixed-precision Q4_K generally yields lower divergence than uniform Q4_0 at similar memory footprints.”

The mechanics, per the Hugging Face docs, explain why. Q4_K organizes weights into super-blocks of 8 blocks of 32 weights, with each block carrying its own scale and minimum, reconstructed as w = q*block_scale + block_min. That per-block rescaling lets weight groups with small dynamic range be represented on a finer grid. Q4_0 applies uniform round-to-nearest over its 32-weight blocks, one grid for everything. The docs also flag Q4_0 as “not used widely as of today,” which makes the preprint’s comparison partly a historical control, though a useful one, since legacy checkpoints still circulate.

The bits-per-weight reference from the cited documentation:

Two caveats travel with the Q4_K result. The preprint hedges with “generally,” implying exceptions across the five architectures tested. And Wikipedia’s independent quality ranking for K-quants over legacy types predates this paper, so the divergence finding corroborates an existing heuristic rather than overturning one.

The checkpoint menu you’re actually choosing from

This decision rarely happens in the abstract, because local deployment runs on GGUF. Per Wikipedia, it is the standard distribution format for quantized LLMs in local inference, natively supported by llama.cpp, Ollama, LM Studio, GPT4All, Jan, and koboldcpp, with tens of thousands of GGUF checkpoints hosted on Hugging Face as of 2026. The ggml spec describes the design goals: single-file deployment, extensible key-value metadata, mmap compatibility, full self-contained information, with v3 adding big-endian support. Community curators like Hugging Face’s GGUF-Models hub require prospective members to bring at least 80 quantized models, which tells you how scheme-dense the ecosystem has become.

The consequence: a model card that says “Q4” is underspecified, and checkpoint choice is where the divergence question actually gets answered, whether deliberately or by default.

A pre-rollout divergence checklist

The following synthesizes the preprint’s framework into a gate you can run before swapping a quant into production. The metrics are standard; the paper does not, in the material reviewed for this piece, name a released tool, so expect to implement the comparison yourself from next-token distributions.

1. Fix the reference. Run the BF16 base model, not a different quant, as ground truth for comparison.
2. Use your own prompts. Divergence on your production distribution is the number that matters; a generic benchmark sample can hide drift on your long-tail inputs.
3. Capture full-vocabulary distributions from base and candidate on the same inputs, then compute JSD and TV per decision point or aggregated.
4. Compare at matched memory. Q4_K versus Q4_0 at similar memory footprints (4.5 bits per weight for Q4_K) is exactly the comparison the preprint ran.
5. Treat accuracy parity as necessary, not sufficient. A matching benchmark score passes the candidate into divergence testing; it does not end it.
6. Reserve the cheapest quants (Q2_K, IQ2_XXS) for workloads that consume only the top prediction. By the paper’s own trend, drift there is largest.

What divergence still doesn’t prove

The authors scope-limit their own work, and the limitation is worth quoting: the findings “do not directly establish correctness, calibration, safety, or user-perceived quality.” A low JSD or TV score means the quant’s distributions sit close to the base model’s. If the base model is poorly calibrated or unsafe, the quant faithfully inherits that. Divergence is a fidelity metric, not a quality verdict, and treating a low number as proof of safety would repeat the original error with a different statistic.

Then there is the evidence itself. This is a single-author preprint whose v2 is days old at the time of writing, observed at abstract level, with no replication and no per-architecture numbers in the abstract. Both headline results are hedged with “generally,” which in a 120-run matrix across five architectures signals non-uniform outcomes, possibly including scheme rankings that flip per model. And arXiv posting is not vetting: administrators do withdraw submissions, as the withdrawal record for arXiv:1011.5746v2 (pulled for plagiarized content) demonstrates. None of this makes the framework wrong. The argmax-blindness critique stands on its own logic regardless of how the numbers settle. It does mean the specific rankings should be treated as one lab’s report until someone else reproduces them.

When is a cheap quant safe to ship?

Split the question by what your workload consumes. If your application reads only the argmax token (fixed-format extraction, classification with a constrained label set, single-answer scoring), then a 4-bit Q4_K checkpoint that matches base accuracy is a defensible default: the paper’s mechanism says your exposure is limited to cases where drift is strong enough to flip the top token, and accuracy testing already samples that. If memory is not the binding constraint, an 8-bit Q8_0 checkpoint should, by the paper’s monotonic trend, sit even closer to base, at the cost of roughly the footprint difference you would expect.

If your application samples, ranks alternatives, or feeds a downstream parser that must tolerate near-ties, accuracy parity tells you almost nothing, and that is where the preprint changes practice. Run JSD and TV against the BF16 base on your own prompt distribution before the swap, set a divergence budget per workload, and prefer Q4_K over Q4_0 at the same memory. Teams that skip this step are not avoiding evaluation cost; they are accepting unmeasured distribution risk in exchange for a benchmark number that, by the mechanism above, cannot see it. The economics tilt the same way: a distribution comparison is a few hours of scripting against base and candidate, while drift surfacing after rollout as inconsistent generations costs a production incident.

The strongest counterweight, stated plainly: everything quantitative here rests on one unreplicated, author-reported preprint observed at abstract level. The decision framework, and the reason argmax scores cannot see distribution change, survive even if the specific numbers do not.

Frequently Asked Questions

What is the difference between Q4_K and Q4_0 quantization schemes?

Q4_K organizes weights into super-blocks of 8 blocks of 32 weights, with each block carrying its own scale and minimum, reconstructed as w = q*block_scale + block_min. That per-block rescaling lets weight groups with small dynamic range be represented on a finer grid. Q4_0 applies uniform round-to-nearest over its 32-weight blocks, one grid for everything.

What metrics should be used to evaluate quantized models beyond accuracy?

It computes “statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between outputs of full-precision and quantized models,” specifically between full-vocabulary predictive distributions at the token decision boundary.