Can Structured Outputs Jailbreak an LLM? Auditing Grammar-Constrained Decoding

Yes, according to one self-reported preprint: a grammar constraint alone, with no jailbreak-crafted prompt attached, can push a code-generating LLM past its safety training. The request is still a plain malicious code request; the grammar does the rest of the work. The CodeSpear paper (arXiv:2606.11817) claims that “simply applying a benign code grammar constraint can effectively jailbreak LLMs,” across 10 popular models and 4 benchmarks, in both locally deployed and API-based setups. Every one of those results is the authors’ own report, and the single preprint is this article’s entire evidence base for them. The detail is unusually complete: the paper names all ten models, from Qwen2.5-Coder-7B through GPT-5, and reports per-model attack success rates.

That combination, a strong claim resting on one preprint’s dense self-reported numbers, is exactly why this article is an audit guide rather than an alarm. If your stack accepts JSON schemas, grammars, or tool definitions from clients, tenants, or runtime-fetched sources, the practical consequence lands on you either way: the schema stops being formatting metadata and becomes untrusted input.

What the preprint actually claims

The paper reports a jailbreak it calls CodeSpear, which exploits grammar-constrained decoding (GCD), the technique engines use to force model output to conform to a target grammar. From the preprint: “We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code.”

The scope, all self-reported by the authors in the same paper: “Through comprehensive experiments on 10 popular LLMs across 4 benchmarks, we show that CodeSpear can effectively bypass the existing safety alignment of both locally deployed and API-based LLMs.” All four benchmarks are named: RMCBench, covering 10 types of malicious coding scenarios, and MalwareBench, covering 6 types of malware-related scenarios, measure safety, while HumanEval and MBPP measure whether the proposed defense degrades ordinary code generation. All ten models are named too: locally deployed Qwen2.5-Coder-7B, Qwen2.5-Coder-32B, Qwen2.5-7B, Qwen2.5-32B and LLaMA3-8B, and API-served GPT-5 and GPT-5-mini through OpenAI, plus MiniMax-M2.5, MiniMax-M2.7 and GPT-OSS-120B through Fireworks AI.

The reported results are specific. Across the five local models, CodeSpear averages an 81.82% attack success rate (the paper), a 26.90-percentage-point gain over unconstrained decoding; across the five API models the average is 67.39%, a 45.39-point gain (the paper). The effect is not uniform, and the paper analyzes the spread: on RMCBench, GPT-5 sits at 55.49% while MiniMax-M2.7 reaches 85.53% (the paper), up from 20.33% without the constraint. The GPT-5 models partly resist by emitting pass statements, a syntactically valid way of doing nothing; when the authors tighten the grammar to forbid pass, GPT-5’s RMCBench rate rises from 55.49% to 70.30% (the paper).

The claim that matters most, and the one to hold most loosely, is the benign-grammar result. The authors frame it as a step beyond prior grammar attacks, which required carefully crafted adversarial grammars; their position is that even off-the-shelf code grammars can be weaponized, and their own implementation ran GCD through llguidance with off-the-shelf Python grammars. That is the novel part of the claim, and it rests on this paper’s own experiments alone. If a follow-up study narrows the result back to crafted grammars, the risk calculus for your pipeline changes considerably, because a crafted grammar requires a more deliberate attacker than a stock Python grammar pulled from a library.

The paper also proposes a mitigation, CodeShield, described as “a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints,” which teaches the model to emit semantically harmless honeypot code when constrained. CodeShield’s source is fully released; the CodeSpear attack code is released only to researchers for controlled use. The preprint encourages the community to adopt CodeShield, and it reports no deployment in any shipping engine.

The mechanism: refusals live outside the grammar

Grammar-constrained decoding works by masking the model’s next-token distribution at each step, zeroing out every token that cannot extend a valid parse under the target grammar. The model can still want to refuse; it simply cannot express the refusal, because a refusal is natural-language prose and the grammar only admits code.

The paper’s own mechanism statement is that CodeSpear excludes natural-language refusals from the valid output space, forcing generation to continue inside the constrained grammar. This is worth sitting with, because it inverts the usual mental model of jailbreaks. Most prompt-level attacks try to talk the model out of refusing: roleplay framing, encoding tricks, multi-turn escalation. This attack does not persuade the model of anything. It deletes the refusal option from the output space the way a form with no “decline” button deletes the option to decline.

That mechanism also explains why the result, if it holds, is architectural rather than model-specific. Safety alignment trains the model to prefer refusal tokens in harmful contexts. A hard constraint on the token stream operates below that preference: no amount of training preference matters for a token whose probability is set to zero. The CodeShield proposal concedes this implicitly; it is a model-side fix precisely because the authors judge the constraint layer able to override the alignment layer.

One boundary on this reasoning: the evidence is the authors’ explanation of their own result. It is a plausible and well-formed mechanism, consistent with how constrained decoding is documented to work, but it is not an independently verified account of what the 10 models were doing internally.

Where grammars come from decides your risk

Not all schemas carry the same trust weight, and this is where the decision actually lives. A schema your team wrote, reviewed, and shipped in version control is a formatting convenience. A schema that arrives at runtime from someone else is input, and input is what attackers shape.

Three provenance tiers, mapped against what the evidence can and cannot say:

The middle row is where most deployments sit, and the remaining inference is narrow. The paper’s threat model is exactly an attacker supplying a grammar through a hosted GCD interface, and its API experiments ran that way against OpenAI and Fireworks AI. The multi-tenant version, a service you operate where tenants or clients define schemas that constrain generation for other users, is not what the paper measures. Reasoning from the paper’s measurement to that architecture is inference, though a short one. I find it convincing enough to act on, because the cost of the controls is low and the failure mode (silent generation of malicious code that passes schema validation) is nasty. But you should know which kind of claim you are acting on.

The third row has the strongest independent backing, from a different attack class, covered below.

Runtime exposure: the constraint path is built in and default for structured generation

The reason a preprint about grammars matters beyond research circles is that constrained decoding is no longer an exotic option. The CodeSpear paper itself notes that GCD “is now supported by many mainstream inference frameworks such as vLLM and SGLang” and “is also exposed by popular commercial platforms including OpenAI and Fireworks AI.” The paper did more than name those platforms: it ran the attack through their hosted grammar interfaces, with GPT-5 and GPT-5-mini on OpenAI and MiniMax-M2.5, MiniMax-M2.7 and GPT-OSS-120B on Fireworks AI.

The deployment picture is broader. Per the XGrammar repository, “It is the default structured generation backend for most LLM inference engines, including vLLM, SGLang, TensorRT-LLM, and MLC-LLM, as well as many other companies.” XGrammar supports general context-free grammars (JSON, regex, custom CFGs) and is marketed on 100% structural correctness of output.

Read that guarantee carefully: structural correctness means the output parses. It says nothing about what the parsed output does. A working exploit chain, a credential stealer, or a plausibly deniable honeypot can all be fully structurally correct. The guarantee and the attack operate on different layers, which is why a green schema-validation check should never read as a safety signal.

Two boundaries remain. The paper does not test llama.cpp’s GBNF grammars, so mapping the attack onto llama.cpp deployments is inference. And “default backend” does not mean every deployment accepts external grammars; many pipelines only ever constrain against schemas the operator wrote. Provenance, not capability, is the exposure question.

The MCP connection: this trust boundary is already bleeding

If the grammar-as-attack-surface idea feels abstract, the Model Context Protocol ecosystem supplies the concrete version. MCP pipelines routinely ingest tool definitions, descriptions, and schemas from third-party servers at runtime, and security researchers already treat that channel as attacker-controlled.

MCP Pitfall Lab enumerates three threat surfaces in MCP-style pipelines: “(i) tool metadata consumed during discovery and planning, (ii) untrusted tool outputs that can carry indirect prompt injections, and (iii) multi-modal inputs, such as images whose extracted text can steer downstream tool calls.” The first surface is the relevant one: metadata, which includes schema-shaped artifacts, parsed by the model before any tool runs.

The Tool Attention paper states the consequence directly: “Because every description token is parsed by the LLM’s reasoning loop, adversarial actors who control a single tool description can inject instructions that hijack the agent without ever being invoked—the Tool Poisoning Attack (TPA).” The same paper cites a cost datum worth knowing, framed there as practitioner-reported rather than peer-reviewed: FinOps audits finding schema tokens responsible for 40–60% of total agent API spend. Client-definable formats carry a bill as well as a risk.

And on measured susceptibility to that adjacent attack class, the MCPTox benchmark reports attack success rates exceeding 60% for GPT-4o-mini, o1-mini, DeepSeek-R1, and Phi-4 (self-reported benchmark results). Be careful with that number: it measures tool poisoning, a prompt-injection-class attack, not grammar-constrained decoding. It belongs in this article as evidence that runtime-fetched tool metadata is a live injection surface against real models. It is not evidence about CodeSpear, and citing it as a grammar-attack success rate would be wrong.

The connection to CodeSpear is structural. Both attack classes exploit the same architectural habit: feeding the model constraint-bearing artifacts (descriptions, schemas, grammars) whose provenance the pipeline never verified. MCP research proves the channel is attacked. CodeSpear, if it replicates, proves the constraint itself can be the payload. Neither source tests the combination.

What you can do before any vendor ships a fix

The paper’s mitigation is CodeShield itself: a research prototype with fully released code, aimed at model developers rather than pipeline operators. For existing deployments, that leaves pipeline-side controls, none of which are measured defenses against this specific attack; they are prudent engineering that shrinks the surface:

1. Allowlist schema sources. Constrain decoding only against grammars and schemas your team authored or explicitly vetted. If a product feature lets API clients or tenants define response formats, that feature now carries a security review burden it probably was not designed for.
2. Run an unconstrained safety pre-pass. Before constrained generation, send the prompt through the model (or a classifier) without the grammar. A refusal that survives in unconstrained space but vanishes under the constraint is exactly the CodeSpear signature. This adds latency and cost; whether that is acceptable depends on how much you trust your schema sources, which is point one.
3. Sandbox anything the constrained output can reach. If generated code executes, renders with privilege, or flows into tool calls, treat it as untrusted output from an untrusted input. The grammar guarantees parseability, nothing more.
4. Log schema provenance. Record where each grammar came from per request. If a jailbreak does surface, provenance logs are the difference between an incident you can scope and one you cannot.

A judgment I would stand behind: if your schemas are all internally authored and code-reviewed, this preprint changes very little for you today; note it and move on. If clients, tenants, or MCP servers can shape your constraints, I would not wait for replication or vendor guidance before adding the allowlist and the pre-pass, because those controls are cheap relative to the failure mode and they defend against the broader, better-evidenced class of metadata injection too.

An audit you can run this week

Convert the above into four questions, answered against your actual deployment:

1. Inventory the constraint paths. Every call site that passes a response_format, guided_decoding, grammar, or schema argument. For each, record who wrote the schema and where it lives.
2. Classify by provenance. Sort each constraint into the three tiers above. For anything client-supplied or runtime-fetched, record what validates it today; the honest answer for most teams is “a JSON parser.”
3. Check the execution path. If constrained output can execute or feed tool calls without a sandbox, a successful constraint attack is a code-execution incident, not an ugly string.
4. Test the refusal-exclusion signature. Take a prompt your model normally refuses, run it once unconstrained and once under your strictest production grammar, and compare. If the refusal disappears under the constraint, you have reproduced the preprint’s mechanism in miniature on your own stack. That test says nothing about whether malicious code follows; it tells you whether your constraint path can prune refusals, which is the load-bearing step.

What this evidence cannot settle

The remaining measurement limits shape how much to spend on this. The per-model spread runs from around 50% attack success on GPT-5 and GPT-5-mini (the paper) to 85.53% on MiniMax-M2.7 against RMCBench, and the safety judgments come from an LLM judge whose agreement with the authors’ own manual checks they put at 87% for ASR (the paper).

What survives regardless of how the preprint fares on replication: the mechanism is sound (hard token constraints sit below trained preferences), the provenance framework stands on its own (runtime-fetched tool metadata is an independently documented injection surface), and the checklist costs little. The grammar you accept is part of your input validation problem now. Whether CodeSpear specifically replicates determines how urgent that is, not whether it is true.

Frequently Asked Questions

How can I test if my grammar constraint removes refusals?

Take a prompt your model normally refuses, run it once unconstrained and once under your strictest production grammar, and compare. If the refusal disappears under the constraint, you have reproduced the preprint’s mechanism in miniature on your own stack. That test says nothing about whether malicious code follows; it tells you whether your constraint path can prune refusals, which is the load-bearing step.

What should I do if clients or tenants can define response formats?

If a product feature lets API clients or tenants define response formats, that feature now carries a security review burden it probably was not designed for.

Does structural correctness of output guarantee safety?

Read that guarantee carefully: structural correctness means the output parses. It says nothing about what the parsed output does. A working exploit chain, a credential stealer, or a plausibly deniable honeypot can all be fully structurally correct. The guarantee and the attack operate on different layers, which is why a green schema-validation check should never read as a safety signal.