Mistral Large 4 FP8 vs NVFP4 vs GGUF: Which Checkpoint Fits Your GPUs

Mistral Large 4 ships in three checkpoint families, and the one with published latency and under-load serving figures is the official FP8 variant: a vLLM recipe splitting the weights across 8 x NVIDIA H200 GPUs with tensor and expert parallelism, with measured latency and throughput, according to the Hugging Face release post. If you have that class of hardware, start there. If you do not, your choice is between the official NVFP4 checkpoint, which has a documented vLLM launch command and a Dell deployment recipe but requires NVIDIA Blackwell GPUs and has no quality benchmark in the release post, and community GGUFs from Unsloth and the llama.cpp team, whose 2 to 6 bit k-quants fit smaller machines (down to a single 512 GB Mac at 2 bits, per the release post) but transfer the quality-verification burden onto you. One more check gates all three: the Mistral Ecosystem Open-Weight License restricts model-as-a-service offerings, and the release post characterizes that clause in one sentence without quoting its scope or thresholds, so read the license text before putting this model behind a hosted API.

One ~1T-parameter MoE, three checkpoint families

Mistral Large 4 (nicknamed “Le Chonk”) is a multimodal, multilingual Mixture-of-Experts model that takes text and images as input and generates text, with roughly 1T total parameters, about 50B active, and a 1M-token context window, per the release post. The post’s prose says 49B active parameters while the model IDs it uses, such as Mistral-Large-4-1T-A52B, indicate 52B, so treat the active count as approximately 50 billion rather than a precise figure. That scale is the entire problem. A trillion total parameters is not something you casually load on a workstation, and the release surface reflects it: the post states that the model “is released in two official variants,” FP8 and NVFP4, adding that “There are also GGUFs created by Unsloth and the llama.cpp team,” and that the model is “distributed under the Mistral Ecosystem Open-Weight License (an MIT derivative that establishes limitations for model-as-a-service offerings).”

The vendor-reported capability numbers are strong on paper: the release post says the model solves 93% of a suite of 40 Cybench challenges drawn from security competitions, and reports 93.3% attack resistance on Lakera’s B3 prompt-injection benchmark. Both figures are self-reported with no independent replication available, so treat them as marketing-grade until third-party evaluations appear. They tell you the model is worth the deployment effort; they say nothing about which checkpoint preserves that behavior, which is the question that actually costs you money.

FP8 on vLLM: the only checkpoint with published latency and under-load figures

The strongest argument for the FP8 checkpoint is not that it is the only documented path (it is not; NVFP4 and GGUF both have documented configurations, and the post publishes measured GGUF throughput) but that it is the only variant with published latency (time to first token) and concurrent-load serving figures (16 requests at a time, 1,000 ShareGPT conversations under load). The release post describes a ready-to-deploy Inference Endpoints recipe:

“The recipe runs vLLM with the FP8 weights split across all eight GPUs using tensor and expert parallelism, and uses the model’s built-in DSpark draft head for faster generation through speculative decoding. It serves up to 131K tokens of context and processes up to 16 requests at a time”

The post then publishes observed results for that recipe: time to first token of about 0.6 seconds on a 1K-token prompt, generation around 65 tok/s, reaching 120–138 tok/s on chat prompts where DSpark’s drafts are accepted. Under load, the post reports about 480 tok/s of total generation across 1,000 ShareGPT conversations. Three details in that recipe matter beyond the headline “8 x H200.” First, it uses both tensor and expert parallelism, which is how you shard a ~50B-active-parameter MoE across eight accelerators without replicating the whole model on each. Second, the DSpark draft head is built into the model itself, so speculative decoding acceleration comes with the checkpoint rather than requiring a separately trained draft model. Third, note the gap between the model’s 1M-token context window and the 131K tokens this recipe actually serves. Eight H200s do not buy you the full advertised context; the KV cache for a million tokens of a model this size is its own capacity problem. If your workload assumes the full 1M window, the published recipe does not demonstrate it. The longest context with a published measurement in the release post is 800K, on the GGUF path described below, and it comes with a steep throughput penalty.

The honest limitation: 8 x H200 is a documented configuration, not a minimum. Whether FP8 fits on fewer H200s, or on H100s or A100-80GB nodes, is not established by the published materials. If your cluster is smaller than eight H200s, you are in extrapolation territory for FP8, and the practical question becomes whether NVFP4 on Blackwell hardware or a GGUF fits your budget instead.

NVFP4: documented for Blackwell, unbenchmarked for quality

NVFP4 is NVIDIA’s 4-bit floating-point format, and the official NVFP4 checkpoint exists to cut the memory footprint roughly in half relative to FP8 (4 bits per weight against 8, before scaling metadata). The format itself has been studied at LLM scale in NVIDIA’s NVFP4 pretraining work, so this is not an exotic encoding. The hardware requirement is stated plainly in the release post: “vLLM supports the high-quality NVFP4 Mistral Large 4 checkpoint, which has a smaller memory footprint and requires NVIDIA Blackwell GPUs.” The post ships a complete vLLM launch command for the checkpoint (tensor parallelism across 8 GPUs, expert parallelism enabled, a 131K-token context cap) and a Dell Enterprise Hub recipe that deploys it on 4 GPUs of a B300 node. So NVFP4 is a documented serving path, but it couples open-weight serving of this model to one vendor’s newest silicon, which is a procurement dependency worth pricing before you commit.

On quality, how Mistral produced the NVFP4 checkpoint has not been disclosed; the post’s “high-quality” label does not say whether a retraining or quantization-aware pass was applied. The relevant evidence if the checkpoint is post-training-quantized without such a pass comes from a study of NVFP4 quantization for low-power edge AI deployment, which found that “NVFP4 without retraining does not match the final retrained accuracy, but it recovers substantial accuracy over conventional FP4 in several models,” concluding that NVFP4 “can provide a useful post-training quantization path for selected architectures.” The phrase that carries the weight is “selected architectures”: the benefit is not architecture-universal, so a trillion-parameter MoE may or may not be one of the architectures where post-training NVFP4 holds up. The release post publishes no quality benchmark for the NVFP4 checkpoint, so the quality question is open regardless of how the checkpoint was made.

On memory, one quantified NVFP4 serving result outside LLM work comes from a long-video-generation pipeline, which reports that “NVFP4 first improves both speed and memory over BF16, and KV-cache quantization further reduces peak memory from 29.7 GB to 19.4 GB with only a modest latency cost” for a 5B video model. That is encouraging for the format generally, but a 5B video generator is not a 1T-parameter MoE inference stack, and these numbers cannot be read as predictions for Mistral Large 4 serving. Treat them as evidence that NVFP4’s memory mechanism works somewhere, not that it works here.

GGUF k-quants: 2 to 6 bits of operator-owned verification burden

The GGUF path is where Mistral Large 4 becomes plausible on a single high-memory machine, because llama.cpp-style k-quants let you dial bits per weight to fit your memory. Per the “Mind the Gap” paper on GGUF quantization, “k-quants generally come in size from 2 to 6 bits per model weight, allowing for a flexible tradeoff of size and model performance,” and the ecosystem is large: the paper counts over 70 thousand k-quant models on the Hugging Face Hub with over 100 million downloads via popular libraries.

For this model, the release post states: “The 2-bit quants run on a single Apple Silicon Ultra computer with 512 GB of unified RAM. The others require distributed inference. You can also run on multiple NVIDIA or AMD cards.” It also reports measured throughput: “Running Q2_K on a single M3 Ultra, we got a prefill throughput of about 90 tokens per second, and a generation speed of 15 tok/s.” On 4 x RTX6000 Pro, the post reports an NVFP4 GGUF at 32K context (prefill 130-590 t/s, generation around 10 t/s) and IQ2_XXS at an 800K context size, where long-context performance drops to 129 tok/s prefill and 3 tok/s generation. In the release post, that 800K result is the only published measurement anywhere near the advertised 1M window, and the throughput collapse at long context is the part to notice.

What the post does not publish is any quality comparison of the quants against the FP8 checkpoint. Throughput numbers tell you the files run; they say nothing about whether a 2-bit quant preserves the capabilities you deployed the model for. That gap is where the “Mind the Gap” paper’s demonstrated attack becomes relevant. The authors show that maliciously crafted k-quants can pass casual inspection: for attacked models, the paper reports, “the benign refusal rate and utility remain close to those of the original models,” while, the paper adds, “upon quantization, the jailbreak score surges to over 90%.” A quant that is subtly wrong, whether through a bad conversion or deliberate tampering, can pass a smoke test while degrading exactly the behavior you care about.

So the GGUF recommendation is conditional. I would use it for evaluation, prototyping, and workloads where a degraded answer costs little, but I would not route customer-facing traffic through a community quant of this model until you have run your own verification pass.

License check: the model-as-a-service clause gates hosted serving

The license is not a footnote here; it changes which serving patterns are available. The release post describes the Mistral Ecosystem Open-Weight License as “an MIT derivative that establishes limitations for model-as-a-service offerings.” That is the whole characterization: the post does not quote the license text, so its precise scope, thresholds, and any revenue or user-count triggers are not established here. Mapped onto three common patterns, that means:

• Internal self-hosting (your team, your GPUs, internal users): the pattern least likely to collide with a model-as-a-service limitation, but confirm against the license text.
• Hosted API for external customers (you sell access to Mistral Large 4 inference): this is the pattern the clause appears to target. Do not build the business before reading the license.
• Embedding in a product (the model powers a feature, customers do not access it as a model): genuinely ambiguous at this level of summary; the classification depends on wording the post does not quote.

The practical instruction is simple: treat the clause as a gating check, not a legal conclusion, and have counsel read the actual license before any hosted-API deployment.

Checkpoint x runtime x VRAM routing table

Read the table as routing logic, not as a guarantee. The FP8 row is the only one with published latency and concurrent-load serving figures on this model; the GGUF row adds published throughput, and no row carries quality evidence beyond the vendor’s own model benchmarks.

A verification checklist for the 4-bit and GGUF paths

If you land on NVFP4 or GGUF, the evidence says quality cannot be assumed, so verify before serving. A bounded checklist that follows from the cited work:

1. Pick reference tasks from your own workload, not generic benchmarks. The edge-AI study’s “selected architectures” finding means published averages may not predict your model-task pair.
2. Diff the 4-bit or k-quant output against the FP8 checkpoint on those tasks. FP8 is the variant with published latency and under-load figures, so it is the natural reference.
3. Verify artifact integrity, not just perplexity. The “Mind the Gap” paper treats quants as an attack surface, which means provenance checks (who produced the file, from which weights, with what tooling) belong alongside accuracy spot-checks.
4. Measure the context you will actually serve. The validated FP8 recipe tops out at 131K tokens, far below the advertised 1M window. The one published long-context measurement, IQ2_XXS at 800K tokens on 4 x RTX6000 Pro in the release post, shows generation falling to 3 tok/s at long contexts, so benchmark at your target length, not at the headline number.
5. Pin the exact artifact you verified. A verification pass covers one specific quant file and runtime version; if you deploy a different build or update the runtime, re-run the check.

What remains unverified

Worth stating plainly, because it bounds every recommendation above: every quality number for this model comes from Mistral’s own release materials, and the cited NVFP4 studies measure edge vision models and a video generator, not this model. The release post reports throughput for the llama.cpp quants but no quality comparison against FP8, and it does not quote the license text behind its one-sentence model-as-a-service summary. Both capability scores (93% Cybench, 93.3% Lakera B3, per the release post) are vendor-reported.

The decision that survives those limits: FP8 on vLLM is the anchor for anyone with H200-class multi-GPU capacity, because it is the only configuration with published latency and concurrent-load serving figures. NVFP4 is the Blackwell-only memory play: documented, but the release post publishes no quality benchmark for it, so benchmark it against FP8 on your tasks before serving. GGUF is the flexible smaller-machine path where you own quality verification end to end. And whichever row you choose, read the Mistral Ecosystem Open-Weight License text before the model goes anywhere near a hosted API.

Frequently Asked Questions

Does the release post provide a quality benchmark for the NVFP4 checkpoint?

The release post publishes no quality benchmark for the NVFP4 checkpoint, so the quality question is open regardless of how the checkpoint was made.