groundy
Infrastructure & Runtime

Qwen 27B in 7.2 GB: When to Trust 2-Bit Quant Compilers

LookAtMyQuant reports Qwen3.8-27B at 7.2 GB with 2.1-bit precision matching original scores, but unreplicated results and missing methodology limit trust for production use.

Published 9 references
On this page11 sections

Qwen3.8-27B at 7.2 GB is worth testing, not yet worth trusting as a drop-in replacement. The LookAtMyQuant repository reports compression from 53.7 GB to 7.2 GB at 2.1 bits per weight, with 401 correct answers out of 540 math problems, matching the original model and beating the same-size Unsloth UD-IQ2_XXS GGUF at 348. Every compiler result is author-reported and unreplicated, and the repository’s published comparison does not name the 540-problem benchmark, evaluation harness, batch size or context length.

That missing methodology matters more than the compression headline. On Apple Silicon, I would trial the build against Unsloth’s same-size file on my own workload, including distribution-level checks, before adopting it. On llama.cpp or CUDA, the published 7.2 GB build is an Apple Silicon MLX release, and the repository’s llama.cpp route is demonstrated on Gemma 4 E2B plans rather than a 27B artifact, so mixed-precision Q4_K_M-style GGUFs remain the better-supported default. If these results reproduce, the limiting factor for small-machine inference starts to shift from fitting weights into memory to verifying what the compressed model has retained.

A useful comparison with an unresolved evidence problem

The comparison is unusually well posed because it controls for file size. Asking which of two approximately 7.2 GB artifacts does more work is much more useful than comparing a 2-bit file with a 4-bit file several gigabytes larger.

The repository reports the following results for its Qwen3.8-27B build. All are author-reported and unreplicated; the original model’s scores are supplied only for the 540-problem math evaluation.

EvaluationLookAtMyQuant, 7.2 GBUnsloth UD-IQ2_XXS, same sizeOriginal model
Unnamed 540-problem math evaluation401/540348/540401/540
HumanEval73.8%67.1%Not given
MMLU78.8%75.0%Not given
GSM8K94.5%92.7%Not given

Source: LookAtMyQuant’s repository, accessed November 22, 2026.

The incumbent is real and actively distributed. Unsloth’s Dynamic 2.0 collection listed unsloth/Qwen3.8-27B-GGUF as updated in November 2026, with 7.53 million downloads. That establishes availability, not accuracy. Unsloth’s statement that “Dynamic 2.0 achieves superior accuracy & SOTA quantization performance” is also a vendor claim, and no independent head-to-head test of these builds appears in the sources reviewed here.

So the defensible reading is narrow: LookAtMyQuant has published a promising same-size comparison, while neither repository popularity nor vendor terminology can substitute for a reproducible evaluation.

What the compiler does with its size budget

LookAtMyQuant is not simply rounding every weight to two bits. According to its method description, the recipe combines:

  • Post-training quantization with trellis codes.
  • Calibration on the model’s own worked math solutions, with digit tokens weighted 10×.
  • Per-row bit allocation guided by Fisher information, rather than one uniform precision everywhere.
  • 4,000 distillation steps, during which every layer is re-quantized once from the original weights.
  • Calibration and distillation coverage described as English prose and chat, code and math.

The distinction between fixed precision and allocated precision matters. A size-budget compiler can spend bits differently across rows, then use distillation to recover behavior lost during quantization. That provides a plausible mechanism for improving on a conventional quant at the same file size. It does not, by itself, demonstrate parity across tasks.

Calibration also freezes assumptions into the artifact. NVIDIA’s quantization explainer describes static quantization as computing parameters once from a calibration dataset and then fixing them for future inference. Here, digit-weighted math calibration may help preserve numerical behavior, but it gives readers no assurance about languages, domains or prompt styles outside the stated mixture. “Code and math” is a broad label, not evidence that the calibration represents your repositories or workloads.

The repository’s stated cost of about $34 in GPU time applies to its 1.23 GB Qwen3.5-4B recipe, not the 27B build. It should not be quoted as the reproduction cost for Qwen3.8-27B.

Apple Silicon users also need to budget beyond the weight file. The repository ships a 1.02 GB, 4-bit DSpine drafter beside the 27B build and reports that it roughly doubles math and code decode speed there. That result is likewise author-reported and unreplicated. In a hypothetical 8 GB machine budget, 7.2 GB of weights plus the 1.02 GB drafter already totals 8.22 GB before the runtime and key-value cache. The KV cache stores context-state information during generation and can add gigabytes at long context, as NVIDIA’s overview explains. A small file therefore does not guarantee a small active-memory footprint.

Why 2-bit parity is a strong claim

The independent literature does not make this result impossible. It does make it an outlier requiring better evidence.

A study of 2-bit recovery with low-rank adaptation summarizes the established pattern: AWQ and GPTQ-class methods are effective at 4-bit precision and can extend to 3-bit, but “accuracy degrades substantially below 3 bits.” Its own Recover-LoRA experiment is more restrained than full parity. After selective 2-bit quantization of gate and up projections in Qwen3-4B, it reports 80–95% accuracy recovery on nine of twelve benchmarks, training LoRA adapters on 10,000 synthetic samples using KL divergence against the teacher’s logits. Partial recovery after that additional training is a different result from an unreplicated claim of original-model accuracy at 2.1 bits per weight.

Other work identifies why recovery can be difficult. RILQ reports that 2-bit LoRA-based quantization-error compensation is constrained by rank sensitivity, even with block-loss optimization. Reporting practice elsewhere is more complete: QuantMoE-Bench, for example, names the lm-evaluation-harness framework it uses to evaluate all tasks. Those are the kinds of specifics missing from the repository’s 27B comparison.

There is precedent for strong method-author results at 2 bits. decoupleQ reports uniform 2-bit post-training quantization close to fp16/bf16 performance for ByteDance’s industrial ASR models. That shows extreme quantization can perform well in a particular setting. It does not independently corroborate a general-language 27B model, and it reinforces the pattern that striking parity claims often begin with the method authors’ own evaluations.

The rational position is therefore neither acceptance nor dismissal. The recipe has mechanisms that could improve retention, while published evidence says sub-3-bit losses are usually substantial and recovery can be incomplete.

KL divergence answers a different question

LookAtMyQuant’s headline fidelity claim is that, on Gemma 4 E2B, its GGUF has 12× lower KL divergence than Q4_K_M at the same file size. That number is author-reported, unreplicated and specific to Gemma 4 E2B. It should not be repeated as a Qwen result or as a general advantage over Q4_K_M.

KL divergence measures disagreement between probability distributions. Lower divergence under a stated test can show that a quantized model’s next-token probabilities remain closer to a reference under those conditions. It does not establish equal downstream accuracy.

The reverse inference also fails. “Accuracy is Not Enough,” an independent divergence-evaluation study, finds that task accuracy can mask substantial distributional differences relative to a BF16 reference. Its evaluation uses full-vocabulary, bounded total variation distance and Jensen-Shannon divergence across 120 configurations. Those metrics examine more of the output distribution than whether the highest-probability answer happens to remain unchanged.

That paper also supplies useful counter-evidence to any simple “fewer bits means proportionally worse behavior” story: for the tested llama.cpp schemes, mixed-precision Q4_K generally produced lower divergence than uniform Q4_0 at similar memory footprints. Bit allocation matters, just as LookAtMyQuant’s Fisher-based allocation suggests.

For a practitioner, the combined lesson is straightforward:

  1. Equal benchmark scores do not prove equivalent behavior.
  2. Lower KL on held-out text does not prove downstream task retention.
  3. A serious comparison should measure both task outcomes and output distributions, with the test corpus and reference model disclosed.

Outside the author’s evaluations, the 7.2 GB Qwen build has not been shown to pass that combined test.

The MoE ladder argues for caution, not transfer

The repository also applies its method to Inkling-2-Mini, described as a 26.4B-physical, 5.4B-active looped mixture-of-experts model. Its six author-reported, unreplicated Apple Silicon MLX builds range from 3.85 GB at 1.18 bits per weight to 12.76 GB at 3.92 bits per weight. The largest build matches the BF16 teacher on every pooled benchmark item, 349/349, according to the repository’s published results.

The lower end is more informative for risk assessment. In the repository’s own table, the 5.38 GB, 1.65-bit-per-weight build scores 174/200 on GSM8K and 117/164 on HumanEval, against the teacher’s 193/200 and 152/164. Its reported all13_KL rises to 0.2773 in the same table. That is a steep retention loss inside the same project claiming dense-model parity at 2.1 bits per weight.

It would be wrong to convert these points into a universal rule that quantization is safe above roughly 2.7 bits and unsafe below it. One MoE family and six builds cannot establish that threshold. They do show why the 2.1-bit Qwen result deserves particular scrutiny rather than extrapolation from the largest Inkling build.

Architecture further limits transfer. QuantMoE-Bench finds that different MoE components require different bit allocations, with shared experts and early MoE layers demanding higher precision. Inkling’s expert-routing and component-sensitivity results therefore cannot serve as direct evidence for dense Qwen3.8-27B, in either direction.

A same-size test you can actually trust

Before replacing a working GGUF, I would require the following protocol.

Pin both artifacts and the runtime. Record immutable digests, repository commits, runtime versions and the exact quant names. A separate local Qwen 3.8 27B study provides a good example: it records its Q4_K_M artifact by digest because a mutable tag alone does not define the experiment. The same discipline prevents a later repository update from silently changing what “the 7.2 GB model” means.

Measure total memory, not just downloads. Record active RAM with the intended context length, plus any drafter. Include the KV cache and runtime overhead. The local-reader paper’s 128,000-token context setting illustrates why weight size alone cannot answer whether a configuration fits. Do not generalize a successful run at one context length to another.

Run the actual same-size comparison. Use the compiler build and Unsloth UD-IQ2_XXS at the stated comparable size. Keep prompts, context, decoding settings and evaluation code consistent. Publish the benchmark name, harness, batch size and context length rather than reproducing the current omissions.

Test your workload separately from public benchmarks. Include representative math, code, chat and prose, plus languages or domains you actually use. The compiler’s calibration description is not evidence of coverage beyond English prose and chat, code and math. Preserve failures and raw outputs so another person can rerun the comparison.

Add distribution checks where logits are available. Measure full-vocabulary divergence against a stated reference on a fixed prompt set, using metrics such as TVD or JSD alongside task accuracy. A reference evaluation may require resources beyond the target machine, but without it, claims of behavioral equivalence remain limited.

Predefine what would block adoption. A small aggregate score gain is not enough if your workload shows new failures, memory exceeds the machine budget at the required context, or distribution checks reveal changes you cannot tolerate. The acceptable tradeoff depends on the cost of those failures, not on bits per weight alone.

When I would switch, and when I would not

On Apple Silicon, I would try the MIT-licensed MLX build for work that can tolerate a failed experiment, provided I could run the same-size A/B and verify active memory. The potential benefit is substantial: a 27B-class artifact small enough to make local experimentation plausible on machines that cannot hold the reported 53.7 GB original.

I would not adopt it for consequential production work solely on the README scores. If a wrong answer or changed behavior creates customer-facing damage, an unreplicated parity claim is not enough. The same caution applies to Unsloth’s superiority language.

On llama.cpp or CUDA, I would stay with a validated Q4_K_M-style configuration. The repository does include a llama.cpp route: gguf_compiler.py builds plans out of llama.cpp’s own tensor types, so the result runs in llama.cpp as it is, and the scripts run inside modal_gguf.py, which builds llama.cpp with CUDA. But that route is demonstrated on Gemma 4 E2B plans, and the published 7.2 GB Qwen build is packed for the Apple Silicon MLX runtime, so no published artifact or score covers the 27B on those stacks. This is a deployment and verification judgment, not evidence that Q4_K_M loses to the compiler at equal size.

What would settle the question is an independent run publishing pinned artifacts, the named math benchmark, harness, batch size, context length, raw outputs and full-distribution measurements, followed by tests on additional models and calibration domains. Until then, the 7.2 GB Qwen build is a credible candidate for evaluation.

Frequently Asked Questions

What is the total memory footprint of the 7.2 GB Qwen build on Apple Silicon?

In a hypothetical 8 GB machine budget, 7.2 GB of weights plus the 1.02 GB drafter already totals 8.22 GB before the runtime and key-value cache. The KV cache stores context-state information during generation and can add gigabytes at long context, as NVIDIA’s overview explains. A small file therefore does not guarantee a small active-memory footprint.

Is the 7.2 GB Qwen build compatible with llama.cpp or CUDA?

On llama.cpp or CUDA, the published 7.2 GB build is an Apple Silicon MLX release, and the repository’s llama.cpp route is demonstrated on Gemma 4 E2B plans rather than a 27B artifact, so mixed-precision Q4_K_M-style GGUFs remain the better-supported default.

What specific details are missing from the repository’s published comparison?

Every compiler result is author-reported and unreplicated, and the repository’s published comparison does not name the 540-problem benchmark, evaluation harness, batch size or context length.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. LookAtMyQuant repositorygithub.comAccessed
  2. Unsloth Dynamic 2.0 collectionhuggingface.coAccessed
  3. NVIDIA model quantization conceptsdeveloper.nvidia.comAccessed
  4. 2-bit recovery with low-rank adaptationarxiv.orgAccessed
  5. RILQarxiv.orgAccessed
  6. QuantMoE-Bencharxiv.orgAccessed
  7. decoupleQarxiv.orgAccessed
  8. Accuracy is Not Enougharxiv.orgAccessed
  9. Local Qwen 3.8 27B studyarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy