Serving GGUF Quants With Transformers Instead of llama.cpp: What Changes

Transformers can now load GGUF checkpoints directly through from_pretrained, which means the quantized files you already download for llama.cpp can run inside a plain PyTorch stack. The practical consequence is stack consolidation: one dependency set can cover prototyping and light serving of the same artifact. What does not change, per Hugging Face’s own announcement, is the recommendation that llama.cpp remains the better engine when efficient local inference is the priority.

What changed: from_pretrained now loads GGUF

Hugging Face’s post, “Transformers now runs llama.cpp quants”, describes the change plainly: “We’re adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop’s memory through the familiar transformers APIs. Pick a GGUF from the Hub, load it with from_pretrained, and start generating on your own machine.”

For the developer this matters to, the workflow inverts. The GGUF file stops being an input to a separate C++ toolchain and becomes an argument to a Python API call. If your environment already carries PyTorch and transformers for fine-tuning experiments or evaluation scripts, the marginal cost of running a quantized checkpoint drops toward zero. There is no separate build, no backend flags, no second binary to keep current.

That is the real news. The rest of this article is about what that consolidation does and does not buy you, because the announcement comes packaged with a benchmark comparison that is easy to misread, and with a vendor caveat that cuts against the more excited interpretation.

One artifact, two runtimes: how GGUF got here

GGUF was never a neutral container. The format specification defines it as “a file format for storing models for inference with GGML and executors based on GGML.” It was developed by Georgi Gerganov, the author of llama.cpp itself, per Hugging Face’s Hub documentation. Format and runtime were co-designed, which is why loading a GGUF file has historically meant running the llama.cpp engine, whether directly or through a frontend.

That coupling is exactly what made GGUF the distribution standard. Per Wikipedia’s GGUF entry, the format was introduced in August 2023 and is now natively supported by llama.cpp, Ollama, LM Studio, GPT4All, Jan, and koboldcpp. The sources pin down the engine for only some of those names: Hugging Face’s post says llama.cpp’s inference engine powers Ollama, LM Studio, and Jan, and Wikipedia notes that Ollama uses llama.cpp as its inference backend. Which engine GPT4All and koboldcpp run is something neither source settles; they are documented only as applications that consume GGUF directly. Groundy’s Ollama vs LM Studio comparison makes the comparable point for that pairing: for most models on most hardware, the bytes doing the matrix multiplication are the same.

Transformers reading GGUF is a path where the artifact stays fixed but the executor changes families, from a GGML-based C++ runtime to the PyTorch stack. The boundary is softer than that framing suggests: to bring performance close to llama.cpp, Hugging Face reuses ggml’s own Metal kernels through the kernels library, so even inside PyTorch much of the matrix work still runs ggml code. What changes is everything around it: the model definition, the memory layout, and the generation loop are transformers’. The second-order consequence is that a single checkpoint becomes consumable by both ecosystems without conversion. Switching serving engines no longer means switching model artifacts. The cost of the switch moves to dependency management and memory behavior, which are the next two questions.

The install-footprint trade

The case for consolidation is operational, not ideological. A maintained llama.cpp setup for serving typically means a source build or pinned release, the llama-bench tool for sanity checks, and a separate server process exposing an OpenAI-compatible endpoint. The transformers path collapses that into the Python environment you already have: per the post, pip install -U "git+https://github.com/huggingface/transformers.git" kernels, load the GGUF file, generate. The documented requirements temper the shorthand: an Apple Silicon Mac, transformers from git main until the next release, and a PyTorch version supported by the published ggml-quantization kernel builds, usually the two latest releases.

The trade is real but asymmetric. PyTorch plus transformers is a large, version-sensitive dependency set, and its install footprint is heavier than a compiled llama.cpp binary by any reasonable measure. Consolidation wins when you were going to carry that Python stack anyway, for training, evaluation, or experimentation, and the llama.cpp toolchain was the extra thing. It loses when llama.cpp was the only thing: a dedicated inference box, a minimal deployment, or hardware where the C++ runtime’s broad backend support is the whole point.

Coverage is the sharper caveat. Per the post, the packed loader currently covers the Qwen3.5 dense and MoE architectures, including compatible Qwen3.8 checkpoints; the packed path is MPS-only for now, and without a compatible quantization kernel the loader falls back to dequantizing the model, which uses more memory. Hugging Face calls adding other architectures relatively straightforward and says coverage will expand gradually. If your checkpoint is not in that set, that is the first thing to check before assuming this path works for you.

What Hugging Face actually measured: tg128 vs generate

The announcement includes a llama.cpp-versus-transformers comparison, and the vendor is unusually explicit about its own methodology:

• The llama.cpp column comes from llama-bench (build 5f55650a7, release b10200, Metal backend from ggml 0.18.0), run as llama-bench -m <file> -p 0 -n 128 -r 3. It reports tg128, per the post: token-generation rate over 128 decoded tokens, averaged across three repetitions, with prompt processing excluded.
• The transformers column, per the same post, is generate producing the same 128 tokens from a 12-token prompt, best of three warmed runs, and it includes prefill.

Per the post, the measurements come from a MacBook Pro M2 Max with 32 GB of unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0, plugged in. Hugging Face states the implication itself: the chart “does not imply identical benchmark conditions, since the Transformers measurement includes prefill while llama-bench reports decode-only throughput.”

Two asymmetries compound here, not one. First, llama-bench reports an average of three runs while the transformers column takes the best of three. A best-of-three on a warmed system measures the favorable tail; an average measures the distribution. Second, prefill is included on one side and excluded on the other. Prefill is a different workload from decode (compute-bound versus bandwidth-bound on most hardware), so folding it into the same per-token number changes what the number means, not just its size.

What the post reports as a result is qualitative: “Transformers is close to llama.cpp across all three checkpoints.” The per-checkpoint rates sit in the chart image rather than the post’s text, so the gap in tok/s is not quotable from the page; anyone citing a precise speed difference from this comparison is reading a chart assembled under two different measurement protocols, not reproducing a number.

A like-for-like benchmark readers can reproduce

The fix is mechanical, and the point of laying it out is that you can run it on your own machine with the same checkpoint in both engines. The procedure below aligns what the published columns leave asymmetric; it is our protocol, derived from the two documented measurement methods, not a vendor specification.

1. Pin both engines. Record the llama.cpp release (the post used b10200) and the transformers, PyTorch, and kernels versions on the transformers side. Version drift is the most common reason cross-engine numbers disagree.
2. Separate prefill from decode on both sides. llama-bench already isolates them (-p and -n are measured separately). On the transformers side, time generate with and without a warmed KV cache, or instrument prefill and decode timing explicitly, so you can report tg128-equivalent decode rate rather than an end-to-end average.
3. Match the run policy. Either average both engines over the same number of repetitions, or take best-of-three on both. Do not mix. The published script in the post sleeps 90 seconds between runs because back-to-back runs decay by 10 percent or more on its hardware; build the same cooling pause in.
4. Match prompt and output. A 12-token prompt with 128 generated tokens, the pair the post used, is a reasonable minimal configuration; also run a realistic prompt length for your workload, because prefill cost scales with it and that is where the protocols diverge most.
5. Fix sampling. Temperature, top-p, and seed differences change generation length and, on some backends, kernel behavior.

Groundy’s MLX vs llama.cpp benchmarking notes add one more discipline worth importing: pair any engine-level microbenchmark with end-to-end timing of real prompts through each engine’s server, warm and cold, since engine measurements can exclude tokenization and sampling time and so benchmark the engine rather than your application’s latency. If what you care about is what a client experiences, decode rate alone will not tell you.

Memory behavior: mmap, alignment, and what PyTorch may not inherit

GGUF’s memory design is a property of the format, which makes it tempting to assume any loader benefits from it. Per Wikipedia, tensor data is aligned by default to a 32-byte boundary “so that weights can be accessed directly through pointers without loading the entire file into RAM, allowing models larger than available memory to be served through operating-system paging.” The alignment survives the move to any runtime. What the transformers loader does on top of alignment is partly documented: per the post, when the weights stay packed on Metal, the ggml-quantization kernel reads them directly in their stored format and “avoids expanding the whole weight matrix before each decode operation.” Whether the loader memory-maps the file and lets the OS page from disk, rather than staging tensors in its own allocations, is not stated.

That distinction is not academic. The paging design belongs to the format, and it is the spec and Wikipedia, not the announcement, that document it: the spec lists “mmap compatibility” among its goals, and Wikipedia describes aligned tensor data allowing a file larger than RAM to serve through operating-system paging. Groundy’s expert-streaming analysis shows both why that works for mostly-resident working sets and why it collapses when every token touches the whole weight file. The transformers path is documented for the case where the checkpoint fits: packed weights stay packed on Metal, so the working footprint tracks the quantized file. The documented fallback is stricter, because without a compatible quantization kernel “the loader falls back to dequantizing the model and uses more memory.” The over-RAM case is a documented capability of the format and is not addressed for the transformers loader. On unified-memory machines, where the OS, the GPU, and the process share one pool, that constraint binds earlier than the file size suggests.

Treat the paging question as something to verify on your hardware: load your target quant through from_pretrained, watch process resident memory against the file size, and check whether loading is lazy. The format gives you the option; the announcement says nothing about whether the transformers loader takes it.

Client compatibility: what actually attaches to each server

Serving implies a client, and the announcement documents both sides.

On the llama.cpp side, the path is documented and current. Per llama.app: “Run llama serve, install the pi-llama plugin and launch Pi. It will automatically discover your local model. No config, no API keys.” The broader GGUF-native ecosystem, Jan, LM Studio, Ollama, GPT4All, koboldcpp, consumes GGUF directly, per the Wikipedia entry, and Hugging Face’s post names Ollama, LM Studio, and Jan as tools powered by llama.cpp’s inference engine.

On the transformers side, the same checkpoint runs behind transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf", which per the post “exposes an OpenAI-compatible API.” The model argument takes the form <model_id>:<filename>.gguf, so a repository holding several quantizations resolves to exactly the file you name. Clients attach by adding a custom OpenAI-compatible provider; the post documents Jan and Pi with Base URL http://localhost:8000/v1 and the same repo:file string as the Model ID, with transformers running the model on the Mac and the client providing the conversation interface. “The same endpoint can be used by other clients that support this API,” though Jan and Pi are the only pairings the post walks through. A --reasoning flag controls thinking modes for chat templates that support them.

Both engines now speak the protocol these clients expect, so client software stops being a reason to pick one runtime over the other. The differentiators move back to install footprint, hardware coverage, and memory behavior.

When llama.cpp still wins, in Hugging Face’s own words

The strongest argument against switching engines comes from the vendor making the announcement: “llama.cpp remains our recommended engine when your priority is efficient local inference. Its dedicated runtime, memory management, and broad hardware support are built around that goal.”

That is a notable sentence for a post announcing GGUF support in transformers, and it should anchor expectations. The post frames the transformers path around Python-side work: experimenting with hooks and custom forward passes, evaluating quantized checkpoints in existing workflows, validating GGUF conversions against the original weights, prototyping decoding ideas, and fine-tuning from a dequantized GGUF. The measured scope is a single Metal configuration on one MacBook Pro with pinned build identifiers, and the packed inference path is MPS-only for now. llama.cpp’s hardware coverage spans far more, and its runtime is purpose-built for exactly the serving workload at issue. None of this means the transformers path is slow; the vendor reports it close to llama.cpp across its three test checkpoints. It means the efficiency case for the C++ engine rests on the vendor’s own stated priority, not on a measurement you can quote precisely.

Practical verdict and what remains open

Consolidate on transformers’ GGUF support when the goal is one Python stack for prototyping and light serving of a checkpoint that fits comfortably in memory: the announcement is real, the API is from_pretrained, and the artifact no longer forces a second toolchain. Keep llama.cpp as the serving engine when efficient local inference or hardware breadth matter, which is the vendor’s own recommendation. And before quoting any cross-engine speed number, from this chart or any other, align prefill treatment, warmups, and repetition counts, because the published columns measure different things and Hugging Face says so.

Three limits bind everything above. The published result is qualitative, “close to llama.cpp” across three checkpoints, with the per-checkpoint values living in a chart image, so any precise speed claim still requires reproducing the measurement under aligned conditions. Client attachment is documented for Jan and Pi against transformers serve, but those are the only pairings the post walks through. And the tested scope is one Metal configuration with pinned build identifiers, so behavior on CUDA, CPU, or other backends awaits independent measurement. The durable change is not a speed verdict; it is that the GGUF file is now decoupled from the GGML runtime, and what used to be a format decision is now a per-workload engine decision you can revisit without re-downloading anything.

Frequently Asked Questions

Which architectures does the transformers GGUF loader currently support?

Per the post, the packed loader currently covers the Qwen3.5 dense and MoE architectures, including compatible Qwen3.8 checkpoints; the packed path is MPS-only for now, and without a compatible quantization kernel the loader falls back to dequantizing the model, which uses more memory.

How do I serve a GGUF model using transformers?

On the transformers side, the same checkpoint runs behind transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf", which per the post “exposes an OpenAI-compatible API.” The model argument takes the form <model_id>:<filename>.gguf, so a repository holding several quantizations resolves to exactly the file you name.