groundy
Agents & Frameworks

Pydantic AI Code Mode Type Checks Are Not Agent Verification

Pydantic AI Code Mode type checks verify interface conformance, not agent correctness. Task verification requires separate outcome assertions after tool side effects.

Published 4 references
A translucent green dinosaur emerges from a keyhole-shaped opening and looks toward a tilted copper target on a warm ivory surface.
On this page9 sections

A generated script that passes Pydantic AI Code Mode’s type-stub check has cleared an interface check, not an agent-correctness check. The vendor documentation describes stubs generated from the agent’s tools, per-script checks, errors returned to the model, and runs recorded in Logfire. It does not document a check that the model selected the right tool, supplied the semantically correct arguments, or produced the intended external state.

The practical response is to keep those concerns separate. Use interface checks before execution, then assert task outcomes after tool side effects. Where your application can retry execution, make side-effecting tools idempotent. Those are recommendations synthesized from the documented boundary and independent failure and testing research, not behaviors the Code Mode documentation claims to provide.

What Code Mode actually checks

Code Mode turns the agent’s tools into Python functions the model can call from generated code. Rather than exposing every tool as a separate model-facing operation, it provides one run_code tool; Agent Sandboxes executes each generated script. Around that execution path, the documented pipeline has four distinct functions:

LayerWhat the evidence establishesWhat it does not establish
run_code interfaceThe model can generate Python that calls exposed tool functions, and Agent Sandboxes runs the script.That the chosen tool or sequence of tools is appropriate for the task.
Type-stub checkEach script is checked against stubs written from the agent’s tools.That validly typed arguments express the user’s intent or produce the right state.
Model-facing errorsCheck errors are sent back to the model.That the model will correct the problem, or that a subsequent run will complete the task.
Logfire run recordEvery run is recorded in Logfire.A task-level verdict, semantic validation, or rollback of side effects.

A fifth layer therefore sits outside the documented check: task-level assertions against the resulting state. That layer is my recommendation, not a named Code Mode feature.

This separation matters because error feedback and observability can make a pipeline look self-verifying. Returning an error gives the model information it may use on another attempt. Recording a run gives developers evidence to inspect. Neither activity, by itself, defines the condition under which the user’s task is correct.

Type correctness leaves decisive failures unresolved

Independent failure-attribution research gives a useful vocabulary for the gap. DeFA: Dependency-Guided Failure Attribution for LLM Agents defines a decisive error step as the earliest erroneous step whose correction would make execution succeed. It assigns a responsible agent and classifies the error using the Pro taxonomy, which includes hallucination and tool parameter or invocation error.

DeFA also represents execution as an event dependency graph. Steps are nodes, protocol relations are combined with semantic dependencies, and the graph records how steps pass and use instructions, results, and state. That is relevant to verification placement: a failure can depend on the meaning and propagation of a value, not merely whether the value fits an interface.

Consider a clearly hypothetical refund(customer_id, amount) call. The script could conform to its stubs while using the wrong customer ID or an amount inconsistent with the request. This example does not establish which mistakes Code Mode’s checker catches. It shows why stub conformance alone cannot establish task correctness.

The vendor’s own example is a counter-signal worth taking seriously. For one question, Pydantic reports that the Code Mode agent was correct in three runs out of three, with a median of 4,680 input tokens. With plain tool calls, the same agent was correct in one run of three and used 111,162 tokens. That supports trying Code Mode when token use and tool-orchestration reliability are problems. It is still one question, three runs, and vendor-judged correctness. It does not turn the stub check into a general verification mechanism.

Not every tool call follows the sandbox path

The checked path also has a coverage boundary. According to the documentation’s selective-sandboxing passage, Code Mode keeps some tools outside the sandbox, including tools that load lazily. The model can call such a tool directly instead of calling it from run_code.

Consequently, a verification design built only around generated scripts can miss part of the agent’s behavior. Outcome assertions should follow the side effect, regardless of whether it arrived through run_code or a direct tool call. That does not imply direct calls are unrecorded or unchecked in every other respect; the vendor page does not settle those questions. It means the run_code pipeline should not be treated as a universal wrapper around every tool invocation.

A snapshot is not a transaction guarantee

State continuity needs equally careful treatment. The vendor says the sandbox-creation function is invoked on the first run_code call of each agent run, concurrent runs receive their own pools, and no connection opens until the model first calls run_code. With Agent Sandboxes, the saved Code Mode state is the ID of a Logfire snapshot. Logfire retains that snapshot for seven days; a conversation resumed after that period starts with a fresh session.

Neither pooling nor snapshot retention defines what happens to external side effects when work is retried. The documentation does not specify rollback or general retry semantics. A seven-day snapshot therefore should not be read as a seven-day transaction, and a fresh session should not be read as a restoration of external state.

If your agent runner or surrounding application can retry a run, design tools so repeated execution is tolerable. For example, a hypothetical invoice-creation tool could accept a stable operation identifier and return the existing invoice when the same operation is presented again. That is an idempotency pattern, not observed Code Mode behavior. The reason to consider it is straightforward: a retry may encounter state mutated by an earlier attempt, and the documented snapshot does not promise to undo that mutation.

Which assertions belong after side effects?

The recommended outcome layer should express what must be true when the task is complete. Depending on the operation, that could mean the requested record exists exactly once, its fields agree with the request, a forbidden transition did not occur, or unrelated state remains unchanged. These are illustrative assertion patterns; their exact form depends on the tool’s contract.

Property-testing research supports preferring invariants over a handful of successful examples, although it does not measure Code Mode. PBT-Bench contains 100 curated property-based testing problems across 40 real Python libraries, with 365 injected semantic bugs designed so that generic random inputs almost never trigger them. The best agent reached 83% bug recall. A newer frontier model and a commercial agent scaffold did not improve on that result, and two of the 365 bugs were never reliably found.

The bounded inference is not that a Code Mode agent will miss exactly 17% of relevant bugs. PBT-Bench concerns agents generating property-based tests in a different setting. Its relevance is that even testing-focused agents leave a residual semantic-bug gap, so generated tests or successful execution cannot be assumed to provide complete outcome coverage.

Assertion strength also matters. The Spec-Driven PBT preprint reports that, across 40 Python projects, each property-based test kills roughly 50 times as many mutants as the average unit test, even after controlling for coverage. The paper attributes that figure to Ravi and Coblenz (2025); it is a cited result, not a fresh measurement by the Spec-Driven PBT authors or a Code Mode benchmark.

I would use that evidence to prioritize a small set of explicit task invariants after side effects, while retaining example-based checks for concrete regressions and known edge cases. The property result motivates the emphasis; it does not establish that the same ratio will hold for agent tools or your application.

Setup facts are not verification facts

The following setup requirements are from the Pydantic documentation, as of 17 November 2026. They describe the documented floors at that snapshot, not forward compatibility or independently tested version support:

Terminal window
pip install 'logfire[agent-sandboxes]>=5.6.0' 'pydantic-ai-harness[codemode]>=0.67.0'

The same setup documentation identifies two configuration failures worth handling directly:

  • Passing a pool instance such as logfire.agent_sandboxes() to monty_sandbox raises UserError. The parameter requires a URL or a function that returns the pool.
  • Using agent sandboxes without a Logfire API key raises LogfireConfigError. The documented remedies are setting LOGFIRE_API_KEY or passing api_key to logfire.configure().

These errors concern whether the execution path is configured correctly. Resolving them says nothing about whether an agent completed its task correctly, and they belong in setup diagnostics rather than an outcome test suite.

What remains unresolved

The vendor documentation does not name the type checker, enumerate which Python errors the check catches, or define rollback and retry behavior beyond the snapshot description. Those are open research questions, not proof that the capabilities are absent. No runtime testing of Code Mode was performed for this analysis.

A team considering Code Mode for consequential workflows should resolve those questions in its own environment with representative invalid scripts and controlled retry scenarios. Even favorable results would leave the central boundary intact: interface conformance and execution diagnostics can help a model produce runnable code, while task verification must determine whether the resulting state is correct.

Frequently Asked Questions

How long does Logfire retain Code Mode snapshots?

With Agent Sandboxes, the saved Code Mode state is the ID of a Logfire snapshot. Logfire retains that snapshot for seven days; a conversation resumed after that period starts with a fresh session.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. Pydantic AI Code Mode Documentationpydantic.devAccessed
  2. PBT-Bencharxiv.orgAccessed
  3. Spec-Driven PBT preprintarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy