groundy
Open Source

Inkling 2 Mini in 5.4 GB: What Extreme Quantization Keeps

Author-reported 5.4 GB Inkling-2-Mini quantization suits short tasks on constrained hardware, but independent evidence warns against replacing 4-bit GGUF for long-horizon ag

Published 4 references
On this page10 sections

The practical answer is conditional: the author-reported 5.4 GB Inkling-2-Mini build is worth trialing for chat and single-shot math or code on memory-constrained machines, but it should not replace a 4-bit GGUF by default for agentic work. Every performance figure for the release comes from its own repository, and independent research on Qwen3.8-27B shows why that distinction matters: a low-bit quantization can nearly match BF16 on a short math benchmark while completing far fewer long-horizon tasks.

That changes the decision. The attraction is not merely a smaller download. It is the possibility of moving 27B-class capability onto machines with 16–32 GB of memory, where the established alternative is often a smaller model at 4 bits. The new bottleneck is verification: you need to know whether the compressed model still completes your tasks, not just whether it fits and produces plausible short answers.

What the 5.4 GB offer actually includes

The LookAtMyQuant repository describes a quantization compiler and several author-reported results. Its code is listed under the MIT license; that statement does not, by itself, establish the licensing of every model artifact.

For Inkling-2-Mini, the repository makes two particularly strong claims:

Those throughput numbers are a useful data point on one Apple system, not an expected-speed guarantee for a 16 GB laptop. They also should not be confused with total memory requirements under every prompt length and serving configuration.

A separate repository result concerns Qwen3.8-27B. According to the project, the same recipe compresses that model from 53.7 GB to 7.2 GB at 2.1 bits per weight, using calibration and distillation on English prose and chat, code and math. This establishes the breadth of the compression claim, but it is not a measurement of Inkling-2-Mini itself. Keeping those artifacts separate matters when assessing the evidence.

A pooled teacher match is not a capability guarantee

The 349/349 result is striking if it reproduces, but a pooled score compresses away the information an adopter most needs: which tasks were included, how long their solutions were, and whether failures cluster on a particular kind of work. There is no third-party replication of these LookAtMyQuant results in the available evidence.

The repository also reports that, on Gemma 4 E2B, its GGUF has 12× lower KL divergence than Q4_K_M at the same file size. KL divergence measures a difference between probability distributions. That can be useful as a fidelity signal, but nothing in the supplied evidence demonstrates that this particular improvement predicts task completion. It should therefore be treated as a screening measurement, not as proof that coding, mathematics or agentic behavior has been preserved. A result on Gemma also cannot validate the separate Qwen and Inkling artifacts.

The same caution applies to comparisons with established quantization projects. The repository does report one author-measured comparison: the KL of its 8.87 GB build sits below Unsloth’s 9.14 GB UD-Q2_K_XL on English, chat and code and across all 13 languages tested. That is a distribution distance reported by the artifact’s own author, not a task-level measurement, and the evidence contains no verified head-to-head between these Inkling builds and a same-size Unsloth or llama.cpp GGUF on real workloads. It cannot settle which artifact is better. What it can settle is that file size, KL divergence and pooled short-task parity answer different questions from “will this finish my workload?”

For model registries, that distinction exposes a practical presentation problem. A listing built around download size and a single benchmark score can make unequal kinds of evidence look interchangeable. A useful quality entry would identify who ran the evaluation, separate short from extended tasks, describe the quantization method and link any independent replication.

Short-answer parity can coexist with a long-horizon cliff

The strongest counter-evidence comes from the NariQ paper, which examines sub-3-bit quantization for long-horizon agentic tasks on the same Qwen3.8-27B family targeted by LookAtMyQuant’s 2.1-bit recipe.

NariQ reports that a 3-bit GSQ-RCO quantization stays within a point of BF16 on AIME: 96.9% against 97.7%. Yet, in results reported by Unsloth and discussed by the paper, it fully solves only 63% as many DeepSWE tasks as BF16. More broadly, the paper states that published Qwen3.8-27B quantizations at or below 3 bits per weight can match BF16 on shorter tasks such as AIME while solving at most 72% as many tasks on DeepSWE, and that on Terminal Bench the reported files at or below 3.0 bits per weight score 64–90% of BF16.

This is not a measurement showing that Inkling-2-Mini fails. It is evidence against using short-benchmark parity as sufficient assurance for any artifact in this compression regime.

A plausible explanation is error accumulation: a small degradation that barely changes a single response can have more opportunities to affect a task requiring many dependent decisions. The reported numbers establish the divergence, however, not that specific mechanism. For a developer, the operational conclusion does not depend on settling the mechanism. If your workload involves extended coding or terminal sessions, evaluate full-task completion rather than extrapolating from math or question-answering scores.

Long-horizon degradation is also not the same thing as long-context-window degradation. DeepSWE and Terminal Bench measure extended task execution; they do not establish how quality changes as the context window fills. If long documents or large repositories are central to your use, that requires a separate test.

Method and architecture change the odds

The NariQ results do not support a blanket rejection of low-bit quantization. The paper reports that NariQ itself, at both 3.0 and 2.5 bits per weight, stays level with BF16 on DeepSWE and Terminal Bench, as well as AIME and RealWorldQA. Those are the paper’s own results, not independent validation of LookAtMyQuant, but they show that failure below 3 bits is method-dependent rather than inevitable.

That makes the recipe part of the product. Calibration data, distillation and the quantization procedure cannot be treated as implementation trivia when two similarly sized artifacts behave differently on extended tasks. “2.5-bit” or “2.1-bit” describes the storage budget, not the quality outcome.

Architecture also matters, although the available evidence requires careful limits. The Mixture of Quantized Experts paper reports that MoE expert weights can be quantized to 3 bits without additional training or calibration data, and to 2 bits with quantization-aware training, producing a 79.6% memory reduction. In its experiments, the MoE model maintained accuracy within 0.3 points down to 3-bit and within 1.82 at 2-bit. The dense comparison preserved accuracy only down to 4-bit and lost more than 2 BLEU points at 3-bit.

Those results came from machine-translation models scored with BLEU, not from modern LLM agent benchmarks. They support an architecture-dependent hypothesis, not a promise about Inkling-2-Mini. The architecture itself is documented: the repository describes Inkling-2-Mini as a 26.4B physical / 5.4B active 3×-looped MoE with 128 experts routed top-8, so the MoE case is the relevant one. What the evidence does not establish is whether expert routing stays stable in the compressed builds. If you are evaluating an MoE artifact, robust expert weights should not be assumed to guarantee stable routing or end-to-end task performance.

Four-bit is a baseline, not an exemption from testing

Staying with 4-bit reduces exposure to the specific sub-3-bit failure pattern documented by NariQ, but bit width alone does not certify the alternative.

The Tetra paper reports that, on Qwen3-4B, a lattice file with int4 value matrices scored about 12 MMLU points below FP16 before repair. Its repair recipe closed only part of that gap.

This is one artifact and one model, and despite its int4 components it is not a 4-bit GGUF: the Tetra files hold about 2.7 bits per parameter over the whole model. It is therefore not evidence that Q4_K_M or 4-bit GGUF generally loses that amount. It does demonstrate why the decision should not be framed as “risky 2-bit versus safe 4-bit”: an artifact whose recipe includes int4 matrices can still sit in the sub-3-bit regime and lose ground. Both sides need task-level evaluation. The difference is that published evidence gives a particular reason to scrutinize long-horizon behavior at or below 3 bits before entrusting it with autonomous work.

A pre-switch checklist that tests the actual risk

  1. Build the evaluation around your real task horizon. Include representative chat, single-response code or math prompts if those are your workloads. For agents, score whether extended tasks are fully solved. A pooled average can hide a workload-specific regression.

  2. Compare against the artifact you would actually ship. Use your current 4-bit GGUF as the comparator, and BF16 where memory permits. Matching a teacher on selected benchmarks is less decision-relevant than beating or matching your existing serving choice on your own evals.

  3. Keep long-context testing separate. Measure quality at the context lengths you expect to serve. Do not treat DeepSWE or Terminal Bench results as evidence about context-window behavior.

  4. Treat the architecture as known and routing stability as unverified. The repository establishes that Inkling-2-Mini is a looped MoE with 128 experts routed top-8, so MoE rather than dense results are the relevant comparison class. Evaluate the complete system rather than inferring stable routing from expert-weight evidence.

  5. Confirm runtime support on the exact artifact. The repository documents an MLX path through its own Apple Silicon runtime, lamq and runtime_mlx, including an explicit generate command for the 5.4 GB file. Loading these packed builds in stock llama.cpp is undocumented. If your serving stack is llama.cpp or another runtime, verify that it loads the file correctly before planning a migration.

  6. Measure memory and speed on your hardware. The M5 Max figures show what the author reports on that system. They do not settle throughput, memory headroom or behavior under your serving settings.

Your situationDefensible choice nowReason
Short chat, math or code on a memory-constrained machine, with time to run evalsTrial the 5.4 GB Inkling-2-Mini build alongside your current 4-bit artifactThe author-reported size and speed are attractive, and the known long-horizon warning is less directly representative of short tasks
Multi-step coding, terminal or other agentic workKeep the established 4-bit recipe, or BF16 where memory allowsPublished sub-3-bit Qwen3.8-27B results show a substantial long-horizon completion risk despite short-benchmark parity
A low-bit method with strong task-level evidence of its ownEvaluate the method rather than rejecting it by bit widthNariQ’s reported 2.5-bit parity shows that the failure pattern is not universal
Your serving stack is stock llama.cpp rather than the author’s MLX runtimeVerify loading before planning a switchThe repository documents an MLX path for these builds through its lamq runtime; llama.cpp loading of the packed files is undocumented

Who should switch, and what would change the answer

I would trial the 5.4 GB build only where a failed experiment is cheap, the work is predominantly short-horizon and your own evals can gate deployment. For a long-running coding or terminal agent, I would stay with the established 4-bit artifact, or BF16 if the machine can accommodate it, while testing the low-bit candidate separately.

The answer would change with independent measurements of the actual Inkling artifacts, especially full-task completion on extended workloads, together with confirmed support in your runtime and satisfactory results on your hardware. Another pooled short-benchmark win or a larger KL improvement would not resolve the central uncertainty, because neither addresses the failure mode that matters most.

Frequently Asked Questions

What are the reported speed and memory figures for the 5.4 GB Inkling-2-Mini build?

The author-reported 5.4 GB build reaches 1,807 tok/s prefill, 75.3 tok/s serial generation and 124.4 tok/s median generation on an Apple M5 Max, with 5.33 GB active RAM. The repository also reports 142.9 tok/s on math and 126.5 tok/s on reasoning.

Does the repository confirm that the 5.4 GB build works with standard llama.cpp?

The repository documents an MLX path through its own Apple Silicon runtime, lamq and runtime_mlx, including an explicit generate command for the 5.4 GB file. Loading these packed builds in stock llama.cpp is undocumented.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. LookAtMyQuant repositorygithub.comAccessed
  2. NariQ paperarxiv.orgAccessed
  3. Mixture of Quantized Experts paperarxiv.orgAccessed
  4. Tetra paperarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy