Running GLM-5.2 on vLLM: MTP Support, Weight Errors, and Throughput

Serving GLM-5.2 on vLLM is, as of August 2026, a speculative-decoding and precision-configuration problem rather than a KV-cache memory problem. The best measured gains on record, 15-19% throughput from training-free multi-token prediction and up to 44% from FP8 in RL rollout settings, come from research on other models, and every GLM-5.2-specific figure circulating in issue threads is self-reported [unverified]. This is what the evidence supports, what it does not, and what you must re-measure yourself.

Why is GLM-5.2 throughput a configuration problem now?

The remaining serving throughput for a large dense-or-MoE checkpoint sits in speculative-decode settings and low-precision kernel paths, because the memory-side bottlenecks that defined LLM serving for the past two years are largely handled by the engine itself. vLLM has matured into the default open-source serving library for transformer LLMs, and its internals are now programmable enough that a plug-in ecosystem exists for manipulating model state directly, per the vLLM Hook paper. When the engine exposes that much surface area, the bottleneck moves to whatever the operator configures badly.

vLLM v0.28.0 landed on Hacker News on 2026-08-30 and renewed operator interest in GLM-5.2 serving recipes [unverified, the release notes and GLM-specific changes were not available in the verified source set for this article]. What follows therefore separates two kinds of claim. Mechanism-level findings from peer-reviewed or preprint research, measured on LLaMA3, Qwen3, and generic dense and MoE models, are cited inline. GLM-5.2-on-v0.28.0 operational claims are marked [unverified] and should be checked against the release notes and the vLLM issue tracker before you act on them.

The practical reframing: if your GLM-5.2 deployment is slow, the first suspects are no longer KV cache sizing and batch geometry. They are whether multi-token prediction is enabled and correctly configured, and whether your precision path is silently degrading output while reporting healthy tokens per second.

Which multi-token prediction method should you enable?

If you cannot modify the checkpoint, training-free probing is the lowest-friction path with measured gains; if the checkpoint ships a native MTP module, use it; the leap-prediction family sits between the two when you control training. Three approaches dominate the current literature, and they differ mostly in what they cost you to adopt.

Native MTP modules. Some checkpoints ship with a dedicated multi-token prediction head trained alongside the base model. The P-MTP paper demonstrates that look-ahead depth is a tunable dimension rather than a fixed model property: it scales MTP depth progressively with a lightweight module to reach high-throughput document parsing. The lesson for operators is that “the model has MTP” is the beginning of the configuration question, not the end. Depth, acceptance thresholds, and verification batching all interact with your serving shape.

L-MTP (leap prediction). L-MTP, a NeurIPS 2025 paper, extends conventional MTP by skipping intermediate positions and predicting non-sequential tokens in a single forward pass instead of predicting strictly adjacent tokens. This is a training-time method. It is relevant if you fine-tune or continue-train GLM-class models, and irrelevant if you serve a frozen checkpoint.

ESP (embedding-space probing). ESP, accepted at ICML 2026, is the method that matters most for operators serving an already-released checkpoint. It probes the model with on-the-fly mask tokens drawn from the model’s own embedding space, producing parallel future-token predictions with no weight modification and no separate draft model. On measured benchmarks it improved acceptance length by 7-11% over LADE on LLaMA3 and 7-8% on Qwen3, and raised throughput by up to 15-19% over the strongest training-free baseline, according to the ESP paper.

Whether vLLM v0.28.0 exposes ESP-class probing, native GLM MTP heads, or both for GLM-5.2 specifically is [unverified]; check the release notes and the model’s serving documentation. The taxonomy above tells you what to look for and what each option costs.

What does a 7-11% acceptance-length gain actually buy end to end?

Materially less than the headline number suggests, because acceptance length is an input to throughput, not throughput itself. A speculative-decoding pipeline drafts candidate tokens and verifies them against the target model; end-to-end speedup depends on acceptance length, draft cost, verification batching, and how much of your workload is decode-bound in the first place.

The honest numbers from the verified set: ESP reports acceptance-length improvements of 7-11% over LADE on LLaMA3, and a separate, smaller end-to-end figure of up to 15-19% throughput over the strongest training-free baseline. Note the structure of those two claims. The acceptance gain is measured against one baseline (LADE); the throughput gain is measured against the strongest available training-free alternative. Neither is measured against a well-tuned native MTP head, and neither is measured on GLM-5.2.

This is where MTP marketing and operator experience tend to diverge. A draft method that doubles acceptance length on paper can deliver single-digit end-to-end gains once verification overhead and batch composition are accounted for, and a method that looks modest on acceptance length can win on throughput because its drafting cost is near zero. ESP’s advantage is precisely that: no draft model, no extra weights, so the 15-19% figure is not paying a drafting tax. When you evaluate MTP settings on your own GLM-5.2 deployment, measure tokens per second at your actual concurrency and context-length distribution. Acceptance length alone will flatter whatever method you just enabled.

Does FP8 beat INT8, and where does W4A8 fit?

For post-training quantization accuracy, the evidence favors FP8 over INT8 across a wide range of networks, and the largest measured FP8 throughput gain, 44%, applies to RL rollout rather than interactive serving. The foundational result is FP8 Quantization: The Power of the Exponent, which found the FP8 format outperforms INT8 on accuracy for post-training quantization across a wide range of networks. The exponent bits buy dynamic range that integer formats spend on precision they rarely need, which is why FP8 has become the default low-precision path for serving as well as training-adjacent workloads.

The throughput ceiling in the verified set comes from FP8-RL, a practical FP8 rollout stack in the veRL ecosystem that supports both vLLM and SGLang as inference engines. Across dense and MoE models, its techniques deliver up to 44% rollout throughput gains while preserving learning behavior comparable to BF16 baselines, per the FP8-RL paper.

W4A8, 4-bit weights with 8-bit activations, is the precision recipe most often mentioned in GLM-5.2 serving discussions, and no verified source in this article’s evidence set measures W4A8 throughput for GLM-5.2 on vLLM [unverified]. What the evidence does say about the FP8 half of that recipe is mechanism-level: FP8-Flow-MoE shows that a quantization-consistent FP8 dataflow, with scaling-aware transpose and fused FP8 operators, can cut explicit cast operations from 12 to 2 and eliminate double-quantization error. Double quantization, casting a tensor to FP8, back to a wider format, and into FP8 again, is exactly the kind of quiet accuracy leak that never appears in a benchmark of tokens per second. If your serving stack’s FP8 path was assembled from generic kernels rather than designed as a dataflow, you may be paying that tax without a metric that shows it.

Which failures never appear in your logs?

Two failure modes dominate low-precision serving, FP8 attention precision collapse and uninitialized weights, and neither produces a logged error, which is why operators lose output quality while their dashboards look healthy.

The FP8 case is now well characterized. P-Cast Precision in FP8 Attention identifies two implementation choices that govern output precision under the attention-sink phenomenon: the KV block iteration order, and the static scaling factor applied to the attention probability matrix P before casting it to FP8. With the wrong iteration order, small P values underflow to zero during the cast and attention output collapses. The paper’s concrete fix: reverse KV-block iteration removes the collapse, and combining reverse iteration with a static scaling factor of S=256 gives a zero-underflow guarantee, according to the P-Cast analysis.

Read that again from an operator’s perspective. Two serving engines can both advertise “FP8 attention,” use the identical format, and produce numerically different outputs because one iterates KV blocks in reverse and the other does not. Nothing crashes. Nothing logs. Perplexity drifts, long-context quality degrades first because attention sinks concentrate the underflow, and the throughput numbers that motivated the FP8 switch keep looking excellent. This is the strongest argument in the verified evidence for treating precision as a per-deployment verification problem rather than a flag.

The second silent failure is closer to GLM-5.2 operations. Operators enabling new architectures on vLLM frequently encounter a “the following weights were not initialized” warning naming indexer.k_norm, and the community diagnosis is that the warning indicates a checkpoint-to-architecture mapping gap rather than a harmless notice [unverified, this diagnosis comes from issue-thread discussion, not from any fetched primary source]. Uninitialized normalization weights mean some fraction of your forward pass is running on random or default values. Inference still completes, tokens still stream, and the degradation shows up only as slightly worse outputs that no serving metric captures. If you see this warning, treat it as a correctness bug until proven otherwise: compare outputs against a reference implementation on a fixed prompt set before you trust any throughput number from that deployment.

Does shared FP8 support across vLLM and SGLang make results portable?

Shared engine support means FP8 recipes port between vLLM and SGLang; it does not mean numerics port, because kernel-level choices like KV block iteration order differ beneath identical format labels. The FP8-RL stack is the useful evidence here: it implements one FP8 recipe across both vLLM and SGLang inference engines alongside FSDP and Megatron-LM training backends, and reports preserved learning behavior versus BF16 baselines. That demonstrates recipe portability in the strong sense, achieved by a team that controlled the full stack and verified the outcome.

What it does not demonstrate is that flipping engines on an existing FP8 deployment is safe. The P-Cast result is the counterweight: FP8 attention output precision depends on implementation details inside the attention kernel, so two engines running the “same” FP8 model can differ numerically. The FP8-Flow-MoE finding points the same direction from the dataflow side: cast placement and fusion choices determine whether double-quantization error exists at all.

For the vLLM-versus-SGLang decision on GLM-5.2 specifically, no verified head-to-head benchmark exists in this article’s evidence set [unverified]. Self-reported comparisons in issue threads and blog posts should be treated as leads to replicate, not as data. The defensible position: pick the engine whose MTP and FP8 configuration surface you can actually verify, because the engine is the smaller variable and your ability to validate it is the larger one. The vLLM Hook work is relevant here too: vLLM’s internals are programmable enough that a motivated operator can instrument acceptance lengths and precision behavior directly rather than trusting aggregated metrics.

What should you verify before trusting your throughput numbers?

Re-measure everything on GLM-5.2 itself, because no figure in the verified research set was produced on that model, and the two biggest numbers in circulation (44% FP8, 15-19% MTP) were measured in settings that do not match interactive serving. A working checklist:

1. Confirm which MTP path your engine exposes for this checkpoint, native head, training-free probing, or none, and measure end-to-end tokens per second at your real concurrency and context-length mix, not acceptance length alone. The ESP results on LLaMA3 and Qwen3 are the reference point for what training-free methods buy; expect GLM-5.2 to differ.
2. Treat the 44% FP8 figure as a rollout-setting ceiling from FP8-RL, and derive your serving expectations from your own FP8 A/B, not from that paper.
3. Audit your FP8 attention path for the P-Cast failure mode: KV block iteration order and the static scaling factor on P. Reverse iteration with S=256 is the configuration with a zero-underflow guarantee per the P-Cast paper; if you cannot determine what your kernel does, that is itself a finding.
4. Investigate every uninitialized-weight warning, including indexer.k_norm, as a correctness bug. The community diagnosis is [unverified], but the failure mode it describes, silent output degradation, is consistent with how these mappings fail.
5. Diff outputs against a reference on a fixed prompt set after any engine, precision, or MTP change. This is the only check on the list that catches the failures the rest miss.

The practical verdict: the levers that matter for GLM-5.2 serving are speculative-decode configuration and FP8 kernel details, and the defaults will not get them right for you. Training-free MTP of the ESP class is the cheapest measured win available (up to 15-19% over the strongest training-free baseline on other models); FP8 stacks larger potential gains but demands per-deployment verification of acceptance behavior and attention precision, because its failures never surface as errors.

The strongest limitation is worth stating plainly. Every number in this article comes from primary research on other models: LLaMA3 and Qwen3 for MTP acceptance, dense and MoE models in an RL stack for FP8 throughput, and kernel-level analyses of generic attention implementations. The GLM-5.2-specific claims that motivated this piece, the v0.28.0 support changes, the indexer.k_norm diagnosis, and any W4A8 throughput figure, remain [unverified] against primary sources. The mechanism transfers; the magnitudes do not, automatically. Measure before you believe anything, including this article.

Frequently Asked Questions

How does the 44% FP8 throughput gain from FP8-RL differ from interactive serving performance?

The 44% gain applies to RL rollout workloads where the system is throughput-saturated and batch sizes are large, allowing full utilization of FP8 kernels. Interactive serving is often latency-bound with smaller batches, so the actual speedup for user-facing endpoints is typically lower and must be measured per deployment rather than extrapolated from training-loop benchmarks.

Why might two vLLM instances running the same FP8 model produce different outputs?

Differences in KV block iteration order within the attention kernel can cause FP8 precision collapse in one instance while the other remains stable. Specifically, forward iteration can lead to underflow of small attention probabilities, whereas reverse iteration combined with a static scaling factor of S=256 provides a zero-underflow guarantee, making kernel configuration a critical variable for numerical consistency.

What is the primary advantage of ESP over native MTP modules for frozen checkpoints?

ESP requires no additional weights or draft models, eliminating the memory overhead and integration complexity of shipping separate MTP heads. This makes it the only viable multi-token prediction option for checkpoints that do not include native MTP support, allowing operators to gain up to 15-19% throughput without modifying the model architecture or retraining.

How does FP8-Flow-MoE prevent the accuracy loss seen in standard FP8 serving paths?

It uses a quantization-consistent dataflow with scaling-aware transposes and fused operators to reduce explicit cast operations from 12 to 2. This eliminates double-quantization error, which occurs when tensors are cast to FP8, back to a wider format, and then to FP8 again, a process that silently degrades accuracy without triggering any logged errors or performance alerts.