For routine monitoring of agent tool inputs and outputs, start with Logfire’s record='failure' mode, switch to 'all' while investigating a specific failure, and use 'metrics' when aggregate validation rates are sufficient. That recommendation follows from the recording behavior in Logfire’s Pydantic documentation, not from a benchmark of agent success under each mode.
The verification rule matters more than the default: a Pydantic record is evidence about data shape, not evidence that the agent selected the right tool or completed the task. Even failure records need careful interpretation. A structured-output failure study found that parse errors can conceal additional schema violations, while separate outcome checks improved results in both vendor-internal Claude Code testing and the TwinCheck preprint. Instrument validation, but verify outcomes separately.
Wire the plugin at the tool boundary
The vendor-doc-sourced configuration pattern imports PluginSettings and attaches it to a model through plugin_settings. This example uses the recommended steady-state mode:
from pydantic import BaseModelfrom logfire.integrations.pydantic import PluginSettings
class ToolArguments( BaseModel, plugin_settings=PluginSettings(logfire={'record': 'failure'}),): ...According to the integration documentation, Logfire implements a new Pydantic plugin API and patches Pydantic to accept that API shape. The documented instrumentation covers three validation methods:
validate_pythonvalidate_jsonvalidate_strings
Its new_schema_validator hook returns decorator factories for those methods, or a tuple containing three None values when recording is off.
That list defines the evidence boundary. It does not establish that every possible way of constructing or validating a model is instrumented. Before relying on the integration for an audit trail, check that your tool boundary actually passes through one of these documented paths. An agent framework can have several ingestion points, and a model having plugin_settings is not, by itself, proof that an unrelated call site was covered.
The configuration details here come from a vendor page fetched on November 17, 2026. Its publication date is unknown, and the fetched evidence does not include runtime verification of the plugin.
Choose a recording mode by the question you need answered
The three modes retain different kinds of validation evidence. The uses below are operational recommendations inferred from those documented behaviors; the fetched sources do not compare their effects on agent task success.
| Mode | Documented behavior | Question it can help answer | What it leaves out |
|---|---|---|---|
'all' | Records all validation events | What passed and failed during a bounded debugging session? | No task-outcome evidence. The fetched page does not specify event payload contents. |
'failure' | Records only validation failures | Which tool-boundary validations were rejected during routine operation? | Successful validations are not recorded, so it cannot reconstruct every pass. |
'metrics' | Records only validation metrics | What do aggregate validation rates look like? | No per-event failure evidence for debugging an individual agent run. |
These behaviors are defined in Logfire’s mode documentation.
I would use 'failure' as the starting point when the operational question is “what is Pydantic rejecting?” It preserves the rejection signal without recording every successful validation event. When a particular bad run needs a sequence of passes and failures, switch the affected models to 'all' for that investigation. If the question is only whether rejection rates are changing, 'metrics' is the appropriate choice, but it cannot support failure forensics on its own.
Recording more events does not repair the underlying verification gap. With 'all', you can have a complete sequence of schema-valid tool arguments and still have the wrong tool choice, a failed downstream request, or an incorrect final state.
There is also an unresolved data-governance question. The fetched page does not document what payload contents 'all' captures, whether sensitive data is redacted, or how Logfire sampling interacts with plugin records. Do not describe this mode as capturing full payloads, removing PII, or guaranteeing a complete sampled audit trail without checking current documentation and your deployment.
The two disable switches have different blast radii
Logfire documents two environment-variable controls:
LOGFIRE_PYDANTIC_RECORD=offdisables the Logfire Pydantic plugin.PYDANTIC_DISABLE_PLUGINS=truedisables all Pydantic plugins.
The distinction is important when telemetry disappears. The first switch is scoped to this integration; the second can affect other Pydantic plugins as well. When recording is disabled, the documented result is no validation wrappers for validate_python, validate_json, or validate_strings, as described on the same API page.
Neither setting answers a separate sampling question. If records are missing, checking these switches establishes whether the plugin was disabled, not whether every emitted record survived the rest of the telemetry pipeline.
Strict and lax validation change what a pass means
A recorded pass also depends on Pydantic’s validation semantics. Pydantic’s validation documentation distinguishes strict mode, where data is not converted, from lax mode, where Pydantic attempts to coerce data to the correct type where appropriate.
Consequently, “valid” does not always mean “the model emitted exactly the expected types.” A lax-mode pass may involve coercion; a strict-mode pass does not. If that distinction matters to a tool contract, keep the strict/lax choice and schema version in your own run or test metadata rather than assuming the plugin record contains enough context to reconstruct it.
Consider a hypothetical agent that emits valid arguments for a refund tool when the request required cancellation instead. Pydantic can accept those arguments under either semantics without addressing the tool-selection error. Similarly, a downstream service can reject or fail to execute a well-formed request. Validation answers whether data satisfied the model’s constraints at that boundary, not whether the agent achieved the intended result.
Failure records can point to the wrong repair
The blind spots are not limited to successful validations. The paper “Why Does Structured Output Fail?” explains that a validation log containing pass/fail status and Pydantic errors cannot, on its own:
- separate a formatting failure from a schema violation hidden behind it;
- determine whether a pattern violation came from its character class or a length ceiling written into the pattern;
- tell whether a prompt or schema change removed a failure or exposed one that was already present.
One reported result makes the first problem concrete: only 48.0% of fenced failures validated once the fence was removed. The remainder concealed a length or pattern violation that the returned parse error never mentioned. That percentage belongs to the preprint’s experimental setup; it is not a predicted failure rate for an arbitrary agent or a measurement of Logfire.
The practical consequence is to avoid treating the first returned parse error as a complete diagnosis. Before changing a prompt or loosening a schema, inspect the output with formatting wrappers removed and check it against the full schema. Compare prompt and schema versions as well. Otherwise, a new error after an edit can look like a regression when the edit merely exposed a pre-existing violation.
This also qualifies the recommendation for 'failure' mode. It is useful for finding rejected data, but the recorded reason for rejection may still require investigation. Use 'all' when the surrounding sequence of validation attempts is necessary to understand that investigation.
Add checks that can fail after validation passes
Outcome verification needs its own assertions. Two sources support that direction, with different scopes and different strengths of evidence.
The Claude Code team reports that, in internal testing around structured file generation, a separate grader checking work against a rubric improved task success by up to 10 points over a standard prompting loop, with the largest gains on the hardest problems. The same verification post describes a bounded recovery policy: if a check is still failing after three rounds, Claude stops and brings in a human. These are vendor-reported internal results and practices, not an independent evaluation of Logfire or proof that three rounds is optimal for every workflow.
TwinCheck supplies a more elaborate mechanism for stateful tool agents. It constructs a trace-grounded counterfactual alternative called a negative twin, then replaces the agent’s proposal only if the twin passes structural checks and a pairwise verifier prefers it in both candidate orders. Structural validity is therefore one gate within the policy, not the final decision.
In its primary analysis of 159 multi-turn BFCL V4 tasks with complete exact-replay pairs, the preprint reports that the complete policy raised GPT-5.6 Sol task success from 45.3% to 58.5%, with a 95% task-bootstrap confidence interval of [8.2, 18.8] and no observed success-to-failure regressions. Those are single-preprint results under specific replay conditions. They support testing counterfactual outcome checks where an incorrect state-changing proposal is costly; they do not establish that the same policy or improvement will transfer to your tools.
A workable verification checklist
The following checklist is a synthesis of the configuration documentation and the outcome-verification evidence, not a tested end-to-end Logfire pipeline:
- Attach
PluginSettingsdeliberately to each tool-boundary model. Use'failure'for routine rejection monitoring,'all'for a bounded investigation, and'metrics'for aggregate rates. - Confirm the ingestion path. Check that validation uses
validate_python,validate_json, orvalidate_strings; do not extrapolate coverage to unnamed call sites. - Preserve interpretation context. Track the schema version and strict/lax choice alongside the run so a pass can be interpreted correctly.
- Treat parse errors as initial evidence, not a complete cause. Check for schema violations hidden behind formatting failures before changing prompts or constraints.
- Assert the selected action separately. A valid argument object still needs a tool-choice or proposal-level check.
- Assert the downstream result separately. Verify the response or intended state change rather than inferring success from schema conformance.
- Bound recovery. A rubric grader can drive retries, but repeated failure needs a stopping rule and human escalation. The Claude Code team’s three-round policy is a documented example to evaluate, not a universal threshold.
- Investigate missing records at the right scope. Check
LOGFIRE_PYDANTIC_RECORD=offbefore the broaderPYDANTIC_DISABLE_PLUGINS=true, then investigate sampling separately if applicable.
Compatibility needs one final check. The Pydantic repository describes V2 as a ground-up rewrite with breaking changes relative to V1 and says V2 includes the latest V1 for incremental upgrades through from pydantic import v1 as pydantic_v1. That upgrade path is not evidence that Logfire’s patched plugin API supports any particular V1 or V2 release. The fetched Logfire page does not state exact version support.
Before production rollout, re-check live version compatibility, runtime behavior, sampling interactions, and sensitive-data handling. Within those limits, the durable division of labor is clear: Logfire’s Pydantic records can show what validation retained or rejected, while task-specific graders, downstream assertions, bounded escalation, and where warranted counterfactual checks must determine whether the agent actually did the right work.
Frequently Asked Questions
Which Logfire recording mode should I use for routine monitoring?
I would use 'failure' as the starting point when the operational question is “what is Pydantic rejecting?” It preserves the rejection signal without recording every successful validation event.

Join the discussion
Share a useful perspective or ask a question about this article.