When LLM Structured Output Fails: What the Pydantic Error Hides

When Pydantic validation keeps failing on your structured outputs, the error message you get back is probably not naming the real problem. In the largest failure category found across 3,997 prompt-enforced generation attempts, markdown-fenced JSON, the authors of a new empirical study report that among 681 fenced failures, only 48.0% pass validation after the fence is removed. The rest conceal a length or pattern violation that the returned parse error never mentions. So a retry loop that reads the surfaced error and re-prompts is, for about half of its most common failure, aimed at a symptom while the binding constraint sits untouched underneath.

The practical consequence: before your next retry, strip the fence and revalidate. If it still fails, you now know the binding constraint is a length or pattern rule the parse error never reported, and the fix is to relax the cap or move that field to a native structured-output mode. If it passes, a repair step (or a prompt that suppresses fences) handles it without touching your schema. Either branch is cheaper than one more round of guessing.

The symptom is not the constraint

The study behind this guide, an arXiv preprint, applies a fine-grained failure taxonomy and a deterministic classifier to those 3,997 attempts across nine OpenAI and Anthropic model/API configurations, using questions from HotpotQA and FinanceBench. The corpus spans 18 configurations of nine models from two vendors, with one baseline schema, two targeted schema variants, and two task domains.

The taxonomy’s central finding is misattribution. Markdown fencing is the most common surface failure, and it looks like a formatting problem: the model wrapped valid JSON in a code block, your parser choked, case closed. But according to the paper, 52.0% of those fenced failures still fail after fence removal, predominantly on field-length violations. The parse error you logged said “invalid JSON.” The constraint that actually bound said “this field is 40 characters over its cap.” Those two problems need different fixes, and the first one was never real.

This is the part that changes how a retry loop should be built. A loop that retries on the surfaced parse error treats the fence as the fault, and a model that re-wraps or re-truncates the same content will keep failing for reasons the loop never sees.

One validity rate, four failure profiles

If you debug structured output with a single number, “this pipeline validates 94% of the time”, the taxonomy shows why that number misleads. The study reports that configurations with overlapping overall failure rates fall into four profiles defined by their dominant failure category. Its documented example: gpt-4o-mini, Sonnet 4.6 and gpt-4o fail at 13.6%, 15.7% and 16.0% with overlapping intervals, under three different dominant categories, pattern, length and schema mirroring respectively.

That distinction is not cosmetic; it determines the fix. A fence-dominant profile responds to a repair step or a prompt change. A length-dominant profile responds to schema relaxation, or to a tighter generation budget, though the budget lever is my inference rather than a remedy the study tested. Treating both as “validation sometimes fails” leads to the same generic retry, which is why the retry keeps not working.

The authors also probe whether the constraints are real. Raising length caps changes which category dominates for the same model, and revalidating the same stored outputs against the relaxed schema shows the caps alone account for the removed length violations. That is a useful design check for your own schema: if a field-length cap is generating failures, you can test whether the cap is load-bearing by revalidating stored outputs against a relaxed variant, without re-running generation at all.

Why the retry loop stalls

Put the pieces together and the stalled retry loop explains itself:

1. The model emits fenced JSON that also violates a length cap.
2. The validator reports the parse error only.
3. The loop re-prompts, possibly with “please output raw JSON.”
4. The model complies on format, and now fails visibly on length, or re-truncates and fails differently.
5. The loop counts one failure rate and reports no progress.

The 48.0% figure from the paper is the ceiling on what fence-stripping alone repairs. The other half needs a schema decision.

One more caution from the same source: in an exploratory prompt ablation, fence-suppressing prompt variants appeared to shift failures toward length violations, a shift the authors read as consistent with exposing violations that were already present rather than creating new ones. If that reading holds, adding “do not use markdown” to your prompt does not reduce work; it moves the failure to a category your logs may not label. The authors themselves flag this ablation as exploratory, so treat it as a warning sign, not a settled result.

The decision: re-prompt, relax, repair, or go native

The four-way decision the reader faces maps onto the failure categories the classifier separates:

The first row is supported directly by the paper’s 48.0% repair rate. The second is supported by the revalidation experiment showing caps alone account for removed violations; the “constrain generation length” fallback in that row is likewise inference, not a paper-tested remedy. The fourth row, moving to a native mode, is where the evidence needs a boundary condition, and it comes from two companion studies.

What native structured-output modes buy, and what they leave behind

The case for going native is real but narrower than vendor announcements suggest. Constrained decoding eliminated all structural failures, raising schema validity from a 78.6–92.9% range to 100% across the tested models, a constrained-decoding study of small LLMs reports. If your binding constraint is structural, the model physically cannot stop emitting invalid syntax, native modes remove that entire failure class.

What they do not remove is the semantic class. The same study finds, in the authors’ words, “content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures (e.g., multi-step function calling) remain CD-resistant.”

The second companion study makes the ceiling concrete. In OrderBench, a deterministic benchmark for restaurant ordering agents, the strongest model kept semantic success near 80% even while achieving 100% schema validity in both prompt-only and JSON-schema modes. Weaker models logged schema-valid unsafe acceptances in the double digits, outputs that passed every structural check and were still wrong in ways that mattered (accepting orders they should have rejected).

So the honest claim about native modes is: they convert structural failures into semantic ones. That is a genuine improvement when your residual failures are structural, and no improvement at all when they are semantic. Before moving a field to native structured output, check which class your failures are actually in, which is exactly what the fence-strip-and-revalidate diagnostic gives you for the structural side.

A field-by-field diagnostic checklist

For a failing validation loop in production, the procedure the evidence supports is:

1. Strip the markdown fence and revalidate the stored output. This separates the 48.0% fence-only failures from the rest, mirroring the fence-strip revalidation the paper ran on its stored wrapped failures.
2. Read the failing constraint from the revalidation error, not the original parse error. The original error never mentions the concealed length or pattern violation the paper finds in the 52.0% of fenced failures that carry one.
3. Classify the binding constraint. Length cap: test whether the cap is load-bearing by revalidating stored outputs against a relaxed variant. Pattern rule: decide whether the prompt or the rule is wrong. Persistent structural failure across prompt fixes: candidate for a native mode.
4. Decide per field, not per pipeline. The study’s corpus is one baseline schema plus two variants; different fields within one schema can bind on different constraints, and the dominant-category profiles shift with task domain and model/API configuration.
5. Verify semantics separately. Even at 100% schema validity, expect residual semantic errors; OrderBench’s strongest model sat near 80% semantic success at full validity.

The paper itself derives four engineering practices from its data, for validator, schema, retry, and evaluation design, and is candid that three are supported while “the fourth, building retry feedback from the fence-stripped validation, follows from them but is untested.” The checklist above leans on the three supported practices; step one’s use inside a live retry loop is the untested part, so trial it before wiring it into automated feedback.

Limits you should carry into the decision

Each rate in this article is author-reported from a single preprint, and the primary study’s deterministic classifier has not been independently verified. Its coverage is two vendors (OpenAI and Anthropic), two task domains (HotpotQA and FinanceBench), 18 configurations, and one baseline schema plus two variants. The authors do release the classifier, raw completions, and annotation record, so the corpus can be relabelled under alternative failure definitions, worth using if your schema’s failure modes look different from theirs.

The semantic-gap evidence also comes from different benchmarks than the primary study: restaurant ordering agents for OrderBench and small open models for the constrained-decoding study. Transferring “semantic success stays near 80% at full validity” to your pipeline is an inference about direction, not a number you can budget against.

The broader context supports the diagnosis-first framing, though. A survey of 51 experienced industry professionals identified 134 concrete output-constraint use cases split between low-level constraints (structured format and length) and high-level semantic and stylistic ones. Length caps of exactly the kind hiding under fenced failures are a first-class industry requirement, not an edge case, which is why the misattribution the parse error commits keeps showing up in real pipelines.

My recommendation, with those limits attached: run the fence-strip revalidation on your next batch of validation failures before changing anything else. It costs one function call against stored outputs, and it tells you whether you have a formatting problem, a schema problem, or a prompt-enforcement ceiling. The validity rate alone never did.

Frequently Asked Questions

What should I do before retrying a failed structured output validation?

The practical consequence: before your next retry, strip the fence and revalidate. If it still fails, you now know the binding constraint is a length or pattern rule the parse error never reported, and the fix is to relax the cap or move that field to a native structured-output mode. If it passes, a repair step (or a prompt that suppresses fences) handles it without touching your schema. Either branch is cheaper than one more round of guessing.

How can I test if a field-length cap is causing validation failures?

That is a useful design check for your own schema: if a field-length cap is generating failures, you can test whether the cap is load-bearing by revalidating stored outputs against a relaxed variant, without re-running generation at all.