MLX vs llama.cpp on Apple Silicon: Which Runtime to Use for Local LLM Inference

Choosing between MLX and llama.cpp on a Mac comes down to three questions the documentation can answer (artifact availability, platform targets, workflow) and one it cannot: raw speed on your specific machine. An earlier version of this article answered all four with rules of thumb, and two were wrong. It claimed MLX was 20-87% faster for models under 14B parameters, and that a roughly 35GB 65B quantization was runnable only in llama.cpp on a 64GB Mac. Both claims are gone from this revision, and the corrections matter because they change what you install and what you believe a given Mac can hold.

The memory claim fails first. Apple Silicon gives the CPU and GPU direct access to one physical memory pool (MLX’s unified memory guide), so llama.cpp’s hybrid CPU+GPU mode changes where layers execute, not how much RAM exists. mlx-lm documents its own handling for models that are large relative to total RAM, including a wired-memory setting for exactly that case (mlx-lm repository). The speed claim fails because the strongest available numbers, Apple’s, compare MLX on an M5 MacBook Pro against MLX on an M4 (Apple Machine Learning Research); llama.cpp is not in that test. What remains is a decision you can actually make, plus a measurement protocol for the part nobody can look up.

One memory pool, not two budgets

MLX’s documentation is unambiguous about the substrate: arrays live in unified memory, and you choose the device when you run an operation rather than when you allocate. The guide’s worked example sends a matmul to the GPU and a chain of small operations to the CPU, with the scheduler inserting dependencies between streams automatically; on an M1 Max, that split ran about twice as fast as keeping everything on the GPU (MLX unified memory documentation). MLX is not a framework that cannot see the CPU. It schedules across both processors over shared arrays.

llama.cpp’s counterpart feature is documented as “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity” (llama.cpp repository), with the split controlled by the --n-gpu-layers count (-ngl), the same knob llama-bench sweeps when testing partial offload (llama-bench documentation). On Apple Silicon there is no separate VRAM to overflow into; there is the same RAM. Splitting layers between CPU and GPU decides where each layer computes, and the CPU-resident layers usually cost you throughput. It does not create memory that MLX cannot address.

What actually decides whether a 65B-class model runs on a 64GB Mac is a budget, and weights are only one line in it. At roughly 4 to 5 bits per weight, 65 billion parameters works out to roughly 33-41GB of weights before anything else; add a KV cache that grows with context length and varies by attention architecture, runtime buffers, and headroom for macOS itself. mlx-lm documents this regime directly: models “large relative to the total RAM available on the machine can be slow,” the package wires model and cache memory on macOS 15 or newer, and when a model fits in RAM you can raise the working ceiling with sudo sysctl iogpu.wired_limit_mb=N, where N should be larger than the model in megabytes but smaller than physical memory (mlx-lm repository).

So the corrected claim reads: a 65B artifact that fits the budget can run under either runtime, provided that runtime supports the architecture and the RAM is genuinely there after cache and overhead. Whether a specific file clears the bar is a load test, not a spec-sheet deduction. Nothing in this article’s source set guarantees a particular 65B quantization fits a particular 64GB machine, in either runtime.

One distinction survives the correction. Mixture-of-experts models cut per-token compute, not resident weight bytes. Apple’s measurements list a 4-bit Qwen3-30B-A3B model (3B active parameters) at 17.31GB of memory on a 24GB machine, nearly the same footprint as a dense 8B in bf16 at 17.46GB (Apple Machine Learning Research). For why expert sparsity changes the compute bill but not the storage bill, see our analysis of MoE expert offloading, which draws on GPU-cluster research rather than Mac measurements.

Which one is faster on your Mac?

Apple’s M5 post is the best performance evidence available here, and it is a single-framework, cross-generation comparison. Apple benchmarked MLX on a MacBook Pro with M5 and 24GB against a similarly configured M4, using 4096-token prompts and 128 generated tokens across six models from a 1.7B dense model in bf16 to a 30B MoE in 4-bit. Generation ran 19-27% faster on M5, tracking the 28% bandwidth increase (120GB/s on M4, 153GB/s on M5). Time to first token improved 3.3x to 4.1x, because prefill is compute-bound and M5’s Neural Accelerators accelerate matrix multiplication; using them requires macOS 26.2 or later (Apple Machine Learning Research).

Those numbers tell you an M5 is a faster Mac for MLX than an M4. They say nothing about MLX versus llama.cpp on either chip, because llama.cpp was not in the test. Shared memory is not an MLX performance advantage either; both runtimes get the same pool and the same bandwidth. This source set contains no controlled, same-machine, same-artifact comparison of the two runtimes, so this revision prints no cross-runtime percentages. Decode is bandwidth-bound and both engines stream the same weights over the same bus; prefill is compute-bound and implementation-sensitive. Any blanket “X is N% faster” claim, in either direction, requires a test this article’s evidence does not contain. The protocol at the end is how you generate one.

Quantization: match artifacts, not bit counts

The old framing that MLX offers only 4-bit and 8-bit is stale. MLX’s core quantize API supports affine quantization at 2, 3, 4, 5, 6 and 8 bits with group sizes of 32, 64 and 128 (defaults: 4-bit, group 64), storing per-group scales and biases. Beyond affine, it supports three block floating-point modes: mxfp4 and mxfp8 with 32-element groups and E8M0 shared scales, and nvfp4 with 16-element groups and E4M3 scales (mlx.core.quantize documentation).

What a conversion actually produces is narrower than what the core supports, which is the practical caveat. mlx-lm’s README documents mlx_lm.convert -q producing a 4-bit model by default, with optional upload to the Hugging Face Hub, and prebuilt conversions live in the mlx-community organization (mlx-lm repository). Whether a given model ships as a 6-bit affine or an mxfp4 artifact is a per-repo question. llama.cpp ships 1.5- through 8-bit integer quantization in the main build (llama.cpp repository).

Same nominal bit count does not mean same artifact or same output. An affine 4-bit with group 64, an nvfp4 block, and a GGUF K-quant at similar file size use different scale machinery and will diverge on hard tokens. When you switch runtimes or step quantization down to fit a memory budget, run your own output checks on your own prompts rather than trusting the label. A preprint analysis of 4-bit versus 8-bit drift we published makes the point sharply: if the top token survives a perturbation, the score is unchanged no matter how much the rest of the distribution moved, so top-token spot checks can miss real damage.

Long prompts: cold starts, warm caches

Prefill cost dominates wall-clock time once prompts get long, and it is paid in full on a cold context. mlx-lm documents three mechanisms that change the arithmetic, each a tradeoff rather than a free win (mlx-lm repository):

• Prompt caching. mlx_lm.cache_prompt writes a processed prompt to a file; later invocations treat it as a prefix, and the Python API supports the same reuse across turns or requests sharing a context. This is the tool for the common agentic pattern of one long document queried repeatedly.
• Rotating fixed-size KV cache. --max-kv-size n caps the cache. Small values such as 512 use little RAM and lose quality; larger values such as 4096 and up use more RAM and preserve it.
• Configurable prefill step size. --prefill-step-size n (default 2048, accepted by generate, chat, server and benchmark) sets how many prompt tokens are processed at once. Smaller steps cut peak memory during prompt processing at some cost to prompt-processing speed.

Caching a long prompt looks like this:

cat prompt.txt | mlx_lm.cache_prompt \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --prompt - \
  --prompt-cache-file mistral_prompt.safetensors

mlx_lm.generate \
  --prompt-cache-file mistral_prompt.safetensors \
  --prompt "\nSummarize the above text."

llama.cpp has its own levers, including a FlashAttention toggle, KV cache dtypes, batch sizes and layer offload, all of them sweepable through llama-bench’s parameter matrix (llama-bench documentation).

The correction here: the previous version of this article declared llama.cpp the universal long-context winner off the back of one third-party report. That does not generalize. With a warm cached context, MLX skips most prefill entirely; cold, the winner depends on the model, the build and the settings. Measure both states.

Python and fine-tuning, or one portable binary

mlx-lm is a Python package first. Installation is pip install mlx-lm, the default model for generation and chat is mlx-community/Llama-3.2-3B-Instruct-4bit, and the API surface is small:

from mlx_lm import load, stream_generate

model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
for response in stream_generate(model, tokenizer, prompt, max_tokens=512):
    print(response.text, end="", flush=True)

Beyond serving, the README lists low-rank and full fine-tuning with support for quantized models, plus distributed inference and fine-tuning through mx.distributed. MLX itself also has Swift and C/C++ interfaces (Apple Machine Learning Research). If adapting weights on the Mac is part of your plan, this is the documented path; llama.cpp’s repository describes inference and serving tools, and training is not among them (mlx-lm repository, llama.cpp repository).

llama.cpp optimizes for portability instead. It is a plain C/C++ implementation with no dependencies, Apple Silicon is handled through ARM NEON, Accelerate and Metal, and the same GGUF file runs across its backends: Metal, CUDA, HIP, Vulkan, SYCL and more, with prebuilt binaries and Docker images on offer (llama.cpp repository):

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF      # download and run
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF    # OpenAI-compatible API server

Both projects ship servers. llama serve is documented as launching an OpenAI-compatible API server; mlx-lm includes a server command of its own (mlx_lm.server), and its API surface is outside what this article’s sources describe, so confirm the endpoint shape in the mlx-lm documentation before pointing client code at it.

How the decision resolves

Every cell below comes from the projects’ own documentation; where the docs are silent, the cell says so.

The rows that usually decide it are the first and the second. If the model you need exists only as GGUF, or your deployment spans a Mac plus Linux boxes with NVIDIA or AMD cards, llama.cpp is the fit. If your work is Apple-only and lives in Python, especially if you fine-tune, mlx-lm is.

The measurement protocol, which we have not run

This article’s contribution is documentation analysis; Groundy ran no hardware tests. This is the protocol that would settle the speed question on your machine, and it is deliberately boring.

Record before comparing anything: Mac model and macOS version; runtime version or commit for both engines; the exact model repository, quantization file and its on-disk size; the chat template applied; input token count; output token cap; and cache state, meaning cold, or warm from a named cache file.

Run the identical task on both runtimes: same prompt, same template, same output cap, fixed sampling settings. Measure five things: total latency cold and total latency warm; time to first token; prefill and decode separated, since a runtime can win one and lose the other; peak memory via memory footprint or pressure; and output acceptability on your actual task, because a faster wrong answer is not a win. Repeat at least five times and report the median and range, not the best run.

llama-bench covers the repeatable half on the llama.cpp side: it runs prompt-processing, generation and combined test types, sweeps parameters such as -ngl, -fa and cache dtypes across combinations, repeats each test five times by default, and averages with standard deviation (llama-bench documentation). Read its own caveat before quoting numbers: measurements exclude tokenization and sampling time, so it benchmarks the engine, not your application’s latency. Pair it with end-to-end timing of real prompts through each runtime’s server, warm and cold. Use --no-warmup when you want cold-start numbers, since the flag exists precisely because warmup runs happen before benchmarking otherwise.

That is the whole decision. Artifact availability, platform reach and workflow choose the runtime; the last question, speed on your Mac, is answered by half an hour of measurement rather than by either project’s documentation.