The evidence is thin, and what exists points toward no. The benchmarks deep-research agents pass with high scores test retrieval from a fixed, curated corpus, not whether the agent can plan a query, filter noise, and ground a claim in the correct primary source across the open web. A cluster of June 2026 arXiv benchmarks is now pressing on that exact gap, and the early results are not reassuring.
Why does passing a curated RAG benchmark say nothing about open-web search?
Curated RAG benchmarks measure whether an agent picks the right passage from a corpus someone already assembled, not whether the agent can assemble the corpus itself.
In a production deep-research task the corpus is not given. The agent has to decide what to search for, which sources to trust, when to stop searching, and how to reconcile sources that disagree. A fixed-corpus eval collapses all of that into a retrieval-and-extraction exercise. A leaderboard score that reads as “the agent understands the documents” is really “the agent understands documents that were pre-selected to contain the answer.” That is a strictly easier problem, and the distance between the two is exactly where deployed agents break.
The DRFLOW benchmark, posted to arXiv in June 2026, is built around that distinction. Its tasks require the agent to identify relevant evidence from scattered, heterogeneous sources and predict the correct action-step sequence, rather than retrieve from a fixed corpus. The design encodes the assumption that matters in practice: the hard part of deep research is not reading, it is finding and filtering, and a benchmark that hands the agent the corpus cannot reach the finding half of the problem at all.
The second-order consequence is that eval scores inflate an agent’s perceived competence at precisely the task buyers care about. A vendor can publish a clean retrieval number; a buyer can deploy the same agent on open-web search and get a different system. The benchmark measures one workload; the deployment exercises the other, and nothing in the gap gets measured unless someone builds an eval for it. DRFLOW is, in part, that someone.
What does DRFLOW measure about agents that find their own sources?
DRFLOW comprises 100 tasks across five domains, with 1,246 reference workflow steps grounded in more than 3,900 sources, scored on seven diagnostic metrics covering factual grounding, step recovery, structural ordering, condition resolution, and personalization.
The authors’ purpose-built agent, DRFLOW-Agent (DRFA), improves over strong baseline agents by up to 10.02% average F1. Read that figure the way the authors do, not the way a launch summary would. It is a relative gain over baselines that were themselves not solving the task; it is not an absolute competence score. The authors explicitly report that substantial room for improvement remains and call complete-and-correct personalized workflow prediction “a challenging frontier for deep research.”
The reason seven metrics matter is that a single aggregate would hide where the agent breaks. Factual grounding, step recovery, and structural ordering can diverge: an agent can recover the right steps in the wrong order, or ground individual facts correctly while missing the workflow they compose, or resolve conditions while ignoring personalization. A retrieval benchmark surfaces none of this because it never asks the agent to sequence, order, or personalize anything. The diagnostic structure is the point. An agent that scores well on grounding and poorly on structural ordering is a different engineering problem from one that scores poorly on both, and only the per-metric breakdown distinguishes them.
The honest read of the current state is narrow. A benchmark-specific agent nudges past generic baselines on a hard task. The gap between “improved over baseline” and “reliable enough to trust unsupervised” is wide open, and the authors are not claiming otherwise.
Do deep-research agent leaderboards predict real-world deployment?
According to the predictive-validity critique, they do not, because aggregate benchmark scores systematically fail to transfer to out-of-distribution settings and no single benchmark touches more than four or five of the dimensions real deployment exposes.
The predictive-validity paper proposes ranking configurations by predictive validity, defined as the correlation between in-sample and out-of-sample rank rather than the in-sample mean, and lays out a twelve-tier measurement apparatus exposing dimensions that HELM and agent-era successors collapse. The mechanism is straightforward. A model that wins a leaderboard optimized for one slice of capability can lose on the slices the leaderboard ignored, and those ignored slices are often the ones that decide whether a deployment works.
Applied to agentic search, this is the structural reason a strong curated-RAG score should not move your priors about open-web performance. The dimensions DRFLOW measures, including source grounding, condition resolution, and personalization, are precisely the dimensions a fixed-corpus eval cannot reach. A leaderboard that omits them ranks agents on a different question than the deployment asks. The first-place model on the eval and the first-place model in production are not guaranteed to be the same model, and the paper’s argument is that the current convention of ranking on in-sample means makes that mismatch invisible.
The contribution is less a new score than a reason to distrust the scores you already have. Anyone reading a vendor’s eval chart is reading the in-sample mean on a benchmark that does not cover deployment dimensions. The paper gives a name, predictive validity, to the property that chart does not have.
How do agents fail when the underlying evidence is thin?
Agents tend to over-read weak evidence and recommend high-stakes interventions with confidence, a failure mode DeXposure-Claw documents in its framing of general-purpose LLM agents for DeFi risk supervision, and one that generalizes to agent-assembled literature reviews.
The literature-review version is quieter than a misfired DeFi liquidation but structurally identical: a confident citation rests on a thin or misattributed source, and nothing in the output flags the weakness. The agent does not emit “I am guessing.” It emits a result that looks sourced, formatted with a link, attributed to an author, and ready to paste into a brief.
A separate uncertainty-decomposition paper attacks the underlying signal problem by separating action confidence from request uncertainty. Across five backbones (GPT-5.1, DeepSeek-v3.2-exp, GLM-4.7, Qwen3.5-35B, and GPT-OSS-120B), the approach improves clarification F1 on ALFWorld-Clarification by 73% over ReAct+UE. The detail that matters for agentic search is the distinction itself: an agent that can tell “I am confident in this action given this query” apart from “this query is underspecified and I am filling the gap with a plausible-sounding result” would, in principle, ask for clarification instead of fabricating one. Shipping agents do not draw that line cleanly, which is why an underspecified search returns a polished citation rather than a question. The fix is not yet standard, and the absence of the fix is the failure mode.
What should you verify in a literature review an agent assembled?
Treat every agent-assembled citation as an unverified pointer until you have checked the source, the claim-to-source mapping, and the query that produced it.
The audit is concrete and short. First, does the cited source actually contain the claim? Agents misattribute. Open the link and locate the passage; if the passage is not there, the citation is decorative. Second, is the source primary? A citation to a primary paper that states a result is different from a citation to a secondary restatement of that result, and agents conflate the two. Third, did the agent signal uncertainty about the query, or return a confident answer to an underspecified question? If the output gives you no way to tell, assume the latter and re-run the query with narrower scope. Fourth, for any number, is there a single primary source, or are several sources being averaged into one confident figure? When sources disagree, surface the disagreement in line rather than collapsing it into a clean number.
That second cost is the real practitioner consequence. An agent-assembled literature review is faster to produce and slower to trust, and the time saved on gathering is only realized if the audit time is paid in full. The broader June 2026 wave is pushing accountability scaffolding in the same direction: execution-bound advisory automation built on an AIBOM-driven CSAF-VEX framework treats agentic outputs as artifacts that need provenance and advisory gating before they drive an action. That is roughly the posture a citation list assembled by an agent deserves. It is not a cite you can trust at face value, and the benchmark that said the agent was good at retrieval is not the benchmark that would have told you so.
Frequently Asked Questions
How does DRFLOW differ from GAIA or SWE-bench-style agentic benchmarks?
GAIA and SWE-bench grade whether an agent reaches the correct end state. DRFLOW scores the intermediate workflow itself: step ordering, condition resolution, and personalization. That diagnostic structure is why DRFLOW can surface an agent recovering the right steps in the wrong order, a failure mode pass/fail scoring hides entirely.
What deployment dimensions does DRFLOW still leave uncovered?
DRFLOW’s seven metrics cover grounding, ordering, and personalization, but standard deployment dimensions like cost per query, latency under load, and off-distribution robustness fall outside its scope. The predictive-validity critique argues no single benchmark covers more than four or five of these dimensions at once. A team optimizing only on DRFLOW stays blind to the axes it does not test.
What does adopting predictive validity require an eval team to change?
Adopting it means holding out an out-of-distribution test set and running enough configurations to compute rank correlation, which costs more compute than a single leaderboard sweep. The metric itself is the correlation between in-sample and out-of-sample rank, not the in-sample mean. Without a held-out set the number cannot be computed at all.
When will agent-assembled citations be trustworthy without a human audit?
The 73% clarification F1 improvement was measured on ALFWorld-Clarification, a household-task benchmark, not on literature search. Transferring the action-confidence and request-uncertainty split to agentic search is unproven. Until that transfer is demonstrated, the human audit remains the only reliable check on an agent-assembled citation.
Does the open-web grounding problem apply to closed-domain agents?
Closed-domain agents that retrieve from a fixed corpus sidestep DRFLOW’s finding problem because the corpus is pre-selected. The pressure applies once the agent must assemble its own sources from heterogeneous material. A legal agent scoped to one case database behaves like curated RAG; a market-research agent pulling from the open web sits in DRFLOW’s failure zone.
