For GLM-5.2 deployments, the serving decision is no longer “which runtime is faster” but “which runtime still has multi-token prediction after you pick a quantization format.” vLLM’s int4 path can drop MTP unless you apply a community patch set, while SGLang’s FP8 and NVFP4 recipes keep EAGLE-style speculative decoding intact. The bottleneck is quantization compatibility, not kernel throughput.
Why does MTP matter for GLM-5.2 latency?
GLM-5.2 is a 744B-parameter mixture-of-experts model built on DeepSeek Sparse Attention with a single multi-token prediction (MTP) layer and a one-million-token context window: 744B from RedlineGT’s consumer setup, architecture from the SGLang cookbook. The MTP head is not a separate draft model; it is baked into the architecture to provide EAGLE-style speculative decoding. Z.ai says IndexShare and an improved MTP layer raise speculative-decoding acceptance length by up to 20% on the GLM-5.2 blog.
That acceptance-length gain translates directly into decode latency. Each accepted draft token is a token the full model does not have to compute from scratch. For an MoE at this scale, the savings are the difference between an unusable chatbot and an interactive endpoint.
The catch is that speculative decoding only helps if the serving runtime can load and run the MTP head under the same quantization as the rest of the model. If quantization strips or disables the MTP head, you keep the memory savings but lose the speedup. That trade-off is what re-ranks vLLM against SGLang for GLM-5.2.
What breaks in vLLM’s int4 path?
Without the patches published on July 5, 2026 by tonyd2wild, vLLM’s int4 path on GLM-5.2 silently breaks speculative decoding. The issue starts in the draft quantization config. The patch README says compressed-tensors starts with an “empty packed_modules_mapping,” so the loader builds the MTP module as unquantized .weight instead of .weight_packed. When DeepSeekMTP.load_weights runs, it KeyErrors on model.layers.78.mtp_block.self_attn.kv_a_proj_with_mqa.weight_packed.
That is a quiet failure mode. The model may still load, but the MTP head is not quantized, and the weight lookup crashes the load path. Teams serving GLM-5.2 from a standard int4 checkpoint on vLLM could reasonably conclude that speculative decoding is unsupported, when the actual problem is a quantization-loader mismatch on a single fused projection.
The same patch set fixes two related int-quantized issues. One is garbage output past 4K-token prompts: chunked prefill casts activations to the Marlin int32 weight dtype once prompts exceed the default chunk size (4096 tokens), according to the patch README. The other is a DeepSeek Sparse Attention buffer sized from max_model_len alone, while the scheduler’s block table can be one block wider due to MTP speculative tokens and scheduler overhang, leading to a size mismatch that crashes the engine at concurrency >= 3.
These are community patches, not necessarily merged upstream, and the version picture is messy. Until the patches land in a tagged vLLM release, “supported” depends on whether you are willing to carry a patched build.
How does SGLang keep MTP working?
SGLang’s documented GLM-5.2 path centers on FP8 and NVFP4, not int4. The SGLang cookbook lists zai-org/GLM-5.2-FP8 as the recommended quantization variant and documents EAGLE-based speculative-decoding presets. For Blackwell, NVIDIA also publishes an NVFP4 build (nvidia/GLM-5.2-NVFP4) that quantizes only the MoE experts’ linear weights and activations to 4-bit, holding accuracy within about one point of the FP8 baseline on several benchmarks, according to the same SGLang cookbook.
That keeps the MTP head inside a quantization format the loader already understands. For teams already committed to FP8 or NVFP4 hardware, this is the simpler path. You trade the memory density of int4 for a quantization format that the GLM-5.2 loader and SGLang’s EAGLE implementation agree on. The memory footprint is larger than int4, but the latency gain from preserved MTP often dominates the cost equation for interactive use.
What does quantization really constrain?
The practical constraint is not “how small can you quantize the weights” but “which quantization format leaves the speculative-decoding path intact.” This is a recurring tension in model serving: quantization buys VRAM and batch size, but model-specific features like MTP heads, fused projections, and custom attention variants are often implemented as assumptions inside the loader. When the quantization framework does not know about those assumptions, it produces a checkpoint that is smaller but functionally incomplete.
The colibrì CPU engine illustrates the same principle from another angle. In RedlineGT’s consumer setup, GLM-5.2’s MTP head must be kept at int8: at int4, draft acceptance collapses to 0-4% (RedlineGT consumer setup) and speculation never engages, while int8 gives 39-59% acceptance (same setup) and 2.2-2.8 verified tokens per forward. That is a CPU inference project, not a GPU server runtime, but the lesson is identical: a more aggressive weight format can silently disable the feature you were quantizing to afford.
For GPU serving, the choice is sharper. vLLM int4 gives the smallest memory footprint but requires community patches to keep MTP alive, and those patches are tied to a specific checkpoint format and loader version. SGLang FP8/NVFP4 gives a supported, documented MTP path at the cost of more memory. If your latency budget depends on speculative decoding, the extra memory is usually worth it.
What should you deploy?
The decision depends on whether speculative decoding is load-bearing for your workload and whether you can carry patched infrastructure.
| Goal | Best fit | Caveat |
|---|---|---|
| Maximum memory reduction, MTP optional | vLLM int4 | Needs tonyd2wild’s July 5 patches; upstream merge status unclear |
| Lowest latency with supported MTP | SGLang FP8 / NVFP4 | Larger memory footprint than int4 |
| Long-context, low-concurrency | SGLang FP8 | Documented EAGLE presets |
| Batch throughput where latency matters less | vLLM int4 post-patch | Once patched, int4 still wins on VRAM and batch size |
If your service-level objective is interactive latency, start with SGLang FP8. The SGLang cookbook documents this path, and FP8 keeps the MTP head intact. You avoid the patched-build risk and the int4 loader edge cases.
If your objective is cost per token at high batch size and you have the engineering bandwidth to maintain a fork, vLLM int4 with the July 5 patches is plausible. The patches fix the MTP load crash, the 4K+ garbage output, and the concurrency-3 DSA crash. But “plausible” is not the same as “supported.” You will be ahead of upstream, and upstream version drift will be your problem.
The broader point is that GLM-5.2 turns the vLLM-versus-SGLang debate inside out. The question is no longer which engine has the faster kernels. It is which engine’s quantization story still includes the model’s built-in speculative decoder. Until vLLM’s int4 MTP path is fixed upstream, SGLang’s FP8/NVFP4 recipes own that question.
Frequently Asked Questions
Does the vLLM int4 MTP failure also affect FP8 or NVFP4 checkpoints?
No. The KeyError on model.layers.78.mtp_block.self_attn.kv_a_proj_with_mqa.weight_packed comes from compressed-tensors using an empty packed_modules_mapping on int-quantized checkpoints. FP8 and NVFP4 load through different quantization configs and dtypes, so the MTP head is not subject to the same packed-weight mismatch.
How do SGLang’s documented EAGLE accept lengths compare to Z.ai’s MTP improvement claim?
SGLang’s cookbook reports EAGLE accept lengths of 4 or more draft tokens, with low-latency runs nearly saturating at 5 to 6 tokens. Z.ai says the improved MTP layer raises acceptance length from 4.56 to 5.47, a 20 percent gain. The two figures are roughly consistent, but SGLang’s numbers are runtime-measured while Z.ai’s are model-level.
Which runtime versions does NVIDIA’s GLM-5.2 model card list, and why does it matter for patched vLLM?
NVIDIA’s model card lists SGLang v0.5.13.post1 or newer and vLLM v0.23.0 or newer. The July 5 patches are community code that may target an older v0.20.2 baseline, so a pinned v0.23.0 build is not guaranteed to include the MTP, Marlin int32, or DSA fixes. You need to verify the patch applies cleanly to your exact vLLM tag before pinning it as a production artifact.
What GPU generations does NVIDIA say GLM-5.2 supports, and does that change the runtime choice?
NVIDIA’s model card supports Blackwell and Hopper. NVFP4 quantization is a Blackwell feature, so Hopper deployments should default to FP8 SGLang or patched vLLM int8 rather than chasing NVFP4. Patched int4 builds may also show different Marlin and DSA behavior across generations, so generation-specific validation is required.
Can the vLLM int4 MTP patches run on a single GPU or consumer hardware?
The published patch set ships from a 4x DGX Spark multi-node setup, not a single-GPU recipe. For consumer hardware, RedlineGT’s colibrì CPU engine uses int8 for the MTP head and achieves 39 to 59 percent acceptance, while int4 acceptance drops to 0 to 4 percent. Running GLM-5.2 int4 MTP on one GPU is not documented in the available sources.
