groundy
Agents & Frameworks

Evaluating Pydantic AI Agents: Should Web Search Be Tested Separately?

Test Pydantic AI web search separately only when integrated runs show recurring retrieval failures. Use end-to-end evals as the success oracle and inspect trajectories to find

Published 4 references
A translucent green resin dinosaur holds a copper-colored egg beside an empty ivory nest. An ivory cuff surrounds its foreleg, while its curled tail loops around a rear ankle. Hard shadows fall across a warm ivory background.
On this page5 sections

Test web search separately when your runs show a recurring retrieval or wiring failure, not simply because WebSearch is a separate capability. For a Pydantic AI agent, I would keep the end-to-end eval as the success oracle, inspect tool-call trajectories to localize failures, and add narrower tests only where they answer a question the integrated eval cannot. Otherwise, a collection of passing component checks can coexist with an agent that still fails the user’s task.

That division of labor fits the documented pieces. Pydantic AI’s overview says Pydantic Evals grades both final outputs and tool-call trajectories. Vercel’s evaluation argument supplies a useful distinction between deterministic tests, one-turn evals, and end-to-end evals, while also warning that individually reasonable steps can produce the wrong outcome. The practical consequence is that a single agent score should start an investigation, not decide which component to change.

When a WebSearch-equipped agent fails, whose failure is it?

Pydantic AI makes the attribution problem concrete by composing an agent from capabilities. Its overview example places Coder(), WebSearch(), and Advisor(...) on the same agent, with WebSearch described as a way to “look up docs and error messages on the web.” More importantly for test design, the documentation says a capability bundles tools, instructions, hooks, and model settings, and that composed capabilities “come apart the way they went together” (Pydantic AI overview).

That gives you plausible test boundaries. It does not tell you which boundary owns a particular failure.

Suppose an agent gives an outdated answer after using web search. Several causes remain open:

  • The model may have made a poor decision about when or what to search.
  • The search step may have returned evidence that did not help answer the request.
  • The capability may not have been wired into the run as intended.
  • The sequence around search may have gone wrong: a reasonable lookup followed by poor filtering, synthesis, or action.

These are diagnostic hypotheses, not failure modes documented for Pydantic AI’s WebSearch implementation. The fetched overview shows WebSearch only as an example capability with a one-line comment. It does not establish provider behavior, result stability, or a supported way to evaluate WebSearch in isolation.

The useful first move is therefore to preserve two levels of evidence: whether the complete task met its requirements, and the tool-call trajectory that led there. According to Pydantic’s overview, Pydantic Evals supports grading final outputs and trajectories against datasets in code or, for online evaluation, a sample of live production traffic. The Pydantic AI repository also describes Pydantic Evals as a separate open-source package and Pydantic Graph as typed control flow for cases where one agent loop is not enough.

A trajectory gives the investigation somewhere to look. It does not, by itself, prove causation. A bad final answer alongside a suspicious search call could still reflect the model’s interpretation of adequate results. Treat the trajectory as evidence for forming the next test, not as an automatic failure classifier.

A useful granularity ladder, with one important caveat

Vercel categorizes tests by how many LLM calls they typically require: deterministic tests use zero, one-turn evals use one, and end-to-end evals use two or more across a whole task (Vercel). This is a vendor’s framing, not an independent measurement of agent reliability, but it maps cleanly onto the decisions a Python team has to make.

A deterministic test is the right candidate for a failure that does not require model judgment: whether an expected capability or handoff is present at a callable boundary, for example. It should not be asked to establish retrieval quality or task success.

A one-turn eval isolates a single decision. That might be a recurring search decision or a recurring interpretation of retrieved evidence. Vercel’s own example is which command an agent runs first. The limitation is central: “Each one can look reasonable on its own, and the agent can still get the wrong outcome.”

An end-to-end eval therefore remains necessary. It lets the agent work through the sequence and grades the final result against the user’s requirements. Vercel also notes that widely used coding benchmarks such as SWE-bench and Terminal-Bench are end-to-end evals, which reinforces the distinction between judging a completed task and judging its component decisions (Vercel).

The caveat cuts against excessive decomposition. If every component passes only its own prompt and fixture, you have not tested the handoffs, accumulated context, or ordering on which the real task depends. Separate tests earn their maintenance cost by identifying a recurring failure class, not by making the eval architecture look tidy.

The capability-to-test-boundary checklist

The following mapping is an inference from Pydantic AI’s composition model and Vercel’s granularity taxonomy. It is not a documented Pydantic testing procedure, and the table does not assume that the current Pydantic Evals API can invoke or mock each capability directly.

Recurring evidence from integrated runsFailure hypothesis to investigateNarrow test worth addingWhat the integrated eval must still establish
A failure occurs before any model or search judgment is neededCapability wiring or deterministic control flowA zero-LLM-call test at the smallest boundary your installed version exposes, with fixed inputs and an expected handoffThat the wired component participates in a successful whole-task sequence
The trajectory repeatedly shows retrieval as the step to investigateSearch retrieval or the inputs supplied to itA one-turn retrieval check with a fixed task context; retain the query and returned evidence used for gradingThat useful retrieval is actually used correctly later in the task
Similar evidence is available, but the model repeatedly makes the same poor decisionModel behaviorA one-turn decision eval with fixed context and repeated runsThat improving the isolated decision improves the completed task
Individual decisions appear defensible, but the final result failsOrchestration or sequence dependenceKeep the investigation at end-to-end level and compare trajectories across passing and failing tasksFinal compliance with the user’s requirements, per task
No failure class recurs below the whole-task levelNot yet establishedNo new component suite; retain integrated grading and trajectory inspectionRegression detection without prematurely assigning blame

Two inputs deserve special discipline.

First, define the task input and the expected user outcome before defining component expectations. Otherwise, a retrieval test can drift toward rewarding plausible-looking results rather than evidence useful for the task. Vercel recommends that eval task sets include common cases, edge cases that trip agents up, and at least one request where the right move is not to act (Vercel). For a search-equipped agent, that last category is a reason to include cases where abstaining or declining an unnecessary action is correct, without assuming that web search should always be invoked.

Second, decide which trace evidence would change your diagnosis. If you cannot articulate what distinguishes a retrieval failure from a model failure in the trajectory, a separate search eval will probably only give you another opaque score. The overview confirms trajectory grading as Pydantic Evals’ scope, but the fetched material does not specify its grader, dataset, or tool-mocking API (Pydantic AI overview).

Nor should you assume that the search provider is the source of run-to-run variation. Vercel’s statement that LLM outputs vary between runs concerns model outputs. None of the fetched sources characterizes WebSearch provider variability. Queries, returned results, and provider behavior may matter operationally, but their stability remains something your own runs would need to establish.

Read the score as sampling evidence, not a verdict

Separate test boundaries make quantitative discipline more important, because a small component eval can appear to convict one layer of the stack. Vercel advises inspecting results per task because overall averages can hide a frequently failing task. Its warning about small samples is direct: with only five trials, 100% means “no failures seen yet,” not “never fails” (Vercel).

That caution applies before comparing conditions as well as after a fix. A model-versus-search comparison based on a handful of runs can easily become a story about which sample happened to fail. The evidence here does not support a universal trial count. It supports making the count visible, reporting per-task outcomes, and avoiding categorical ownership claims from a tiny difference.

Independent research gives a further reason for restraint, although from a different domain. In an author-reported evaluation of seven frontier LLMs on long-horizon resource allocation and commitment coordination, several agents underperformed a simple rule-based baseline, while only the leading agent consistently surpassed it. The authors also state: “With only two world seeds and one run per model–seed pair, the stability of the observed rankings under broader sampling remains uncertain” (arXiv evaluation).

This was not a Pydantic AI or web-search evaluation, so it does not predict your agent’s failure rate. Its relevant contribution is methodological: rankings can be fragile under limited sampling, and a more elaborate agent is not automatically better than a simple baseline. If a deterministic rule or non-agent workflow can handle a recurring task, that is a legitimate comparison rather than an embarrassing fallback.

Finally, expect an eval’s job to change. Vercel says evals saturate as models and products improve: a saturated eval still catches regressions but cannot tell you what to improve next (Vercel). At that point, adding harder whole-task cases may be more informative than splitting an already-passing search component into ever smaller checks.

What to verify before committing to the suite

The recommendation is deliberately narrower than “test every capability separately.” Keep end-to-end grading as the arbiter of user success. Use trajectories to decide whether a recurring failure justifies a deterministic wiring test or a one-turn retrieval or decision eval. Require the narrower test to improve diagnosis or regression detection; otherwise, keep the evidence at integrated level.

Before implementing that plan, verify the parts the available documentation does not settle: the current Pydantic Evals API for datasets and graders, whether your installed version provides a suitable isolated or mocked search boundary, and the actual behavior of your chosen search provider. The fetched excerpts do not show that these capabilities are absent; they simply do not document them. Independent Pydantic AI failure reports are also missing from the evidence, so the failure classes above should guide investigation rather than be treated as measured frequencies.

If no isolated boundary is available, that does not prevent useful evaluation. Keep the integrated eval, inspect its trajectories, and record the unresolved attribution honestly. A narrower score you cannot validly construct is less useful than a whole-task result whose limits you understand.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. Pydantic AI's overviewpydantic.devAccessed
  2. Vercel's evaluation argumentvercel.comAccessed
  3. Pydantic AI repositorygithub.comAccessed
  4. arXiv evaluationarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy