When an agent starts emitting malformed tool calls on a self-hosted vLLM deployment, the reflex is to blame the model: re-prompt, fine-tune, or shop for a bigger checkpoint. The evidence assembled here points somewhere else first. The clearest account comes from a single vendor’s engineering blog, with no independent replication, so treat the specific numbers accordingly: Ubicloud reports that serving Kimi K2.6 on vLLM 0.28 produced the expected tool-call schema only 92% of the time, and that the failure rate moved with serving flags and vLLM versions while the model weights stayed constant. If that account holds, the bottleneck for self-hosted agents shifts from model selection to serving configuration, and a routine vLLM upgrade can silently change agent behavior.
The practical consequence: before touching prompts or models, identify which layer is producing your symptom, because four distinct failure modes look identical from the outside and are fixed in four different places.
Match the symptom to the layer first
Intermittent tool-call failures on vLLM cluster into four observable symptoms, and each points at a different component:
Malformed JSON or schema misses. The model emits a tool call that fails validation: invalid syntax, wrong types, missing required fields. This is the failure Ubicloud measured. On Kimi K2.6 with vLLM 0.28, “2 out of every 25 attempted tool calls failed,” per their writeup. Schema misses at this rate are the symptom most directly addressed by constrained decoding, the serving-layer mechanism that forces output to conform to a grammar.
Silent no-tool responses. The agent simply never calls the tool, returning prose instead. No parse error, no exception, nothing in the logs that looks like failure. This is the most dangerous symptom, and the Tool Suppression study documents it as a reproducible production pattern: “multiple evaluated open-weight models cease invoking tools when Tool Calling and Structured Output constraints are simultaneously enabled.” If your agent goes quiet on tools after you enabled structured output, the constraint machinery itself is a suspect.
Missing arguments that still pass your eval. The call parses, the tool name is right, but a required parameter is absent or hallucinated, and your benchmark still scores it as correct. The action-class diagnostic paper shows this is common: xLAM posts 71.8% aggregate accuracy but only 19.5% Gold Action Recall on missing-parameter scenarios, 64.5 pp below size-matched peers, “while BFCL’s state grader still marks these cases as PASS.” If you only see this failure in production and never in evaluation, your grader is the layer to fix.
Calls to nonexistent tools. The model invokes a function that is not in the provided list rather than refusing. The same diagnostic paper found explicit refusals are rare across general-purpose families: in missing-function scenarios, gpt-oss-20b records a miss_func Δ of −38.5 pp and GLM 5.1 of −32.5 pp, where Δ is Gold Action Recall minus state-graded accuracy, meaning the state grader passes many trajectories in which the model never refuses. Failures surface as calls to held-out functions with fabricated arguments instead of clean refusals. This one leans more toward model behavior than serving configuration, though the serving stack still shapes how those fabricated calls surface.
What the vendor evidence actually shows
Ubicloud’s account is the spine of the configuration-first argument, so it is worth laying out precisely, with its limits attached. Serving Kimi K2.6 on vLLM 0.28, they measured 92% schema accuracy. Their initial fix was a custom constrained-decoding implementation. Then, per the blog: “After vLLM added this feature in v0.25, we switched to using this feature from upstream for Kimi and DeepSeek. Unfortunately not long after, we found out that our tool-call accuracy score dropped back to 91%!” The same capability, implemented differently upstream, regressed the metric by a point below their own baseline.
Their resolution: “This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.”
Three caveats belong next to that 100% figure. First, it is vendor-reported with no independent benchmark; Ubicloud operates inference infrastructure and has an interest in the story. Second, the trajectory of 92%, then 91%, then 100% spans three vLLM versions, so every number is version-scoped and will drift with later releases. Third, the claim covers “newer Kimi and DeepSeek models” and says nothing about other families, which turns out to matter.
Even taken at face value, though, the episode establishes the central point: tool-call accuracy moved by eight to nine points based purely on serving-layer changes, with identical model weights. That is not a model problem.
When the fix is the bug
Constrained decoding works by compiling your JSON schema into a finite-state machine and masking every token that would violate the current FSM state. The Tool Suppression paper describes the mechanism directly: with response_format enabled, “JSON Schema constraints are compiled into grammar-based token masks that render tool-call tokens unreachable during decoding,” with logits of negative infinity assigned to nonconforming tokens so they cannot be sampled.
That mechanism explains both sides of the story. Applied to the tool-call payload, it guarantees schema-conformant arguments, which is the fix Ubicloud wanted. Applied to the whole response when structured output and tool calling are enabled together, it can make the tokens that initiate a tool call unreachable, so the model never starts one. The study found complete suppression across all tested open-weight models under that combined condition, while the closed-source reference, GPT-5.4-mini, maintained stable tool execution under all tested conditions. Whether that gap reflects serving-stack differences, training, or both, the paper does not settle; the safe takeaway is that open-weight deployments under simultaneous constraints need explicit testing, not assumption.
Ubicloud’s own experience corroborates the risk from the other direction. After vLLM enabled constrained decoding by default, they report: “For example, gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported issues after having this feature enabled by default,” because constrained output forced token choices those models otherwise would not make. The same flag that restored 100% schema accuracy for newer Kimi and DeepSeek models degraded behavior elsewhere. This is the counter-evidence that rules out a blanket recommendation: constrained decoding is a per-family decision, not a global one.
Which models benefit, and which break
The evidence supports a family-scoped matrix, with one important empty cell:
| Model family | Constrained decoding effect | Evidence basis |
|---|---|---|
| Kimi (K2.6 and newer) | Reported fix: 92% to 100% schema accuracy on vLLM 0.31 | Ubicloud, vendor-reported |
| DeepSeek (newer) | Reported fix: 100% schema accuracy with the 0.31 flag | Ubicloud, vendor-reported |
| gpt-oss-20b | Reported regressions when enabled by default; miss_func Δ (GAR − Acc) of −38.5 pp | Ubicloud; arXiv 2609.00949 |
| Qwen 3.6 | Reported issues with default-on constrained decoding | Ubicloud, user reports |
| GLM 5.1 | Reported issues with default-on constrained decoding; miss_func Δ (GAR − Acc) of −32.5 pp | Ubicloud; arXiv 2609.00949 |
| GLM 5.2 | Not covered by the cited reports | Untested; nearest data is GLM 5.1 |
The empty row matters if you are running or evaluating GLM 5.2 on vLLM: nothing in this evidence base tests it. GLM 5.1 is in the “reported issues” column, which is grounds for caution with the newer checkpoint, not grounds for assuming the regression carried over or was fixed. Test it yourself before enabling the flag.
A vLLM upgrade is a behavior change
The thread connecting Ubicloud’s three data points is version drift. In their account, v0.25 introduced upstream constrained decoding and regressed their Kimi/DeepSeek accuracy to 91%. Version 0.28 measured 92% on Kimi K2.6 without it. Version 0.31 shipped the server-side flag that restored 100% on those families, and somewhere in that window the feature became default-on in a way that broke gpt-oss-20b, Qwen 3.6, and GLM 5.1 users. Any of these transitions could happen inside a routine dependency bump, with no change to your model, prompts, or tool schemas, and your agent’s behavior shifts underneath you.
Two disciplines follow. Pin the vLLM version in production and treat upgrades as behavior changes requiring a tool-call regression test, not as invisible maintenance. And record which structured-output and constrained-decoding flags are active per deployment, since defaults have already changed once in this version range and may change again. I would not roll a vLLM minor version into a production agent fleet without a schema-accuracy eval in the rollout gate; the documented case for that caution is the 91% regression that appeared only after switching implementations.
Is your eval lying to you?
Even a correct fix can look wrong, or a broken stack can look fine, depending on the grader. The action-class diagnostic shows pass/fail state graders like BFCL’s mark missing-parameter calls as PASS, which is exactly the failure class most likely to reach production undetected. Argument-level metrics such as Gold Action Recall expose what the state grader hides: xLAM’s missing-parameter GAR sits 64.5 pp below size-matched peers, a deficit that aggregate accuracy obscures entirely.
For a baseline sense of how much headroom exists at the model layer, ToolFailBench evaluated 1,000 single-turn tasks across finance, medicine, law, cybersecurity, and real estate, and found Grok-4.3 with the highest Clean Tool-Use Rate at 86.33%, followed by Grok-4-1-Fast-Reasoning at 84.11% and Qwen2.5-32B-Instruct at 82.68%. Even the best model in that benchmark fails clean tool use in more than 13% of tasks. Model selection is not irrelevant; it sets a ceiling. But a serving misconfiguration can put you well below whatever ceiling your model offers, and no model upgrade will recover a loss caused by a grammar mask.
An operational checklist
Ordered by what to check first when tool calls degrade on vLLM:
- Capture the symptom precisely. Schema-validation failures, silent no-tool responses, and wrong-or-missing arguments have different causes. Pull raw model output before any parser error handling swallows it.
- Pin and record the vLLM version. Every behavioral claim in this article is scoped to v0.25 through 0.31. Know exactly which version regressed or fixed what for you.
- Audit active constraints. Check whether constrained decoding or structured output (
response_format) is on, including defaults, and whether tool calling and structured output are simultaneously active. That combination is the documented Tool Suppression trigger. - Scope constrained decoding by model family. Enable it for newer Kimi and DeepSeek deployments where the vendor reports full schema accuracy; leave it off, or test extensively before enabling, for gpt-oss-20b, Qwen 3.6, and GLM-class models where regressions are reported.
- Grade at the argument level. Use Gold Action Recall or equivalent per-argument checks rather than pass/fail state graders, and include missing-parameter and missing-function scenarios in the eval set.
- Re-run the tool-call eval after every vLLM upgrade. The 92% to 91% regression appeared from an implementation swap, not a model change. Assume the next upgrade can do the same.
What this evidence cannot settle
The honesty requirements here are substantial. The quantitative spine (92%, 91%, 100%) comes from one vendor’s blog with no independent replication, and the academic papers study constraint mechanisms in their own stacks, not Ubicloud’s deployment. None of the sources assembled here quantifies the overhead of constrained decoding on a production workload. The only statement of that kind in this evidence base is Ubicloud’s, attributing to XGrammar that “this runtime enforcement adds negligible performance overhead,” which is a vendor claim rather than an independent measurement. Overhead remains a reasonable expectation from any FSM-masked decoding path; measure it on your own workload. The evidence does not establish which --tool-call-parser or chat-template settings fix specific models, so template-level triage remains undocumented territory. And GLM 5.2 appears nowhere in the cited reports; the nearest anchored data point is GLM 5.1, in the regression column.
What the evidence does support is the triage order. When tool calls break on vLLM, the published record says the serving layer changed the outcome while weights stayed fixed, repeatedly, across versions. Check the configuration before you blame the checkpoint, scope any constraint flag to the families it demonstrably helps, and make your eval capable of seeing the failures you are trying to fix.
Frequently Asked Questions
Which vLLM version introduced the server-side flag for constrained decoding?
This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.
What happens when Tool Calling and Structured Output constraints are enabled together?
multiple evaluated open-weight models cease invoking tools when Tool Calling and Structured Output constraints are simultaneously enabled.
How does the BFCL state grader handle missing-parameter tool calls?
xLAM posts 71.8% aggregate accuracy but only 19.5% Gold Action Recall on missing-parameter scenarios, 64.5 pp below size-matched peers, “while BFCL’s state grader still marks these cases as PASS.”
Join the discussion
Share a useful perspective or ask a question about this article.