How to Evaluate AI Agents on Long Tasks: Public Benchmarks vs Your Own Environment

The preprint “Scaling Verifiable Environments for Long-horizon Work Agents” (arXiv 2610.04906) reports that its WorkForge pipeline can instantiate 16.7K verifiable environments across 40 professional domains. That number is author-reported, unreplicated, and derived from a training pipeline rather than a deployment evaluation harness. It is still worth your attention, but not for the reason the headline suggests.

Here is the practical finding this article rests on: if you are deciding whether a coding or work agent can be trusted with a 12-to-24-hour task, a public leaderboard score is weak evidence either way, and the WorkForge preprint does not change that. The real decision is narrower. Use a pinned public long-horizon benchmark for coarse comparability against other teams’ results, and gate your actual rollout on a small custom environment that mirrors your workflow, with checkers you have audited, a sandbox you control, measured reset reliability, and multi-rollout consistency reporting.

Why single-session leaderboard scores don’t predict multi-hour work

The case against leaderboard-driven rollout decisions does not rest on vibes. It rests on three documented gaps between what public benchmarks measure and what a long-horizon deployment requires.

First, the established frameworks were not built for this. A 2026 review of production agent evaluation states it plainly: “Existing evaluation frameworks for large language models — including HELM (Liang et al., 2022), MT-Bench (Zheng et al., 2023), AgentBench (Liu et al., 2023), and BIG-bench (Srivastava et al., 2022) — are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production”. That is not a criticism of those frameworks; it is a statement of scope. A single-session score tells you about single-session competence.

Second, the failures that matter at long horizons are qualitatively different. RetailBench ran seven contemporary LLMs through a simulated retail business over a 180-day evaluation horizon and found that only a small subset of models survived the full horizon at all, and even the strongest runs “remain substantially behind” a privileged oracle policy in net worth and sales. The failure analysis is the part worth quoting: agents show “incomplete evidence acquisition, surface-level decision making,” and the paper notes that “agents often fail to maintain coherent strategies across long interactions, especially when local errors compound through delayed consequences and reshape future state distributions”. That compounding-error dynamic is exactly what a short benchmark cannot see.

Third, even the benchmarks that look most impressive are narrower than their scores suggest. The OSWorld 2.0 paper notes that Claude Opus 4.8 reaches 83.5% on OSWorld-Verified, a score it says suggests desktop computer use is largely solved, but immediately qualifies this: “the tasks behind this number are short and narrow, rarely spanning more than one or two applications, and reward completing self-contained actions rather than sustaining long, connected workflows.” An 83.5% on short self-contained tasks is not evidence about a 12-hour connected workflow. Meanwhile, the Meta-Agent Challenge calibrates what the multi-hour regime actually looks like in current research: 12-hour time budgets for AIME, GPQA, and LiveCodeBench, and 24-hour budgets for SWE-Bench and Terminal-Bench. If your rollout decision concerns tasks in that range, that is the evaluation regime your evidence needs to come from.

Four axes that decide the question

The choice between an off-the-shelf benchmark and a custom environment is not ideological. It decomposes into four axes, each with documented evidence behind it.

Task horizon. Single-session tasks favor public benchmarks; that is what they measure well. Once your target work spans 12 to 24 hours, the evidence above says you are in a regime where public scores transfer poorly and compounding errors dominate.

Checker verifiability. A benchmark is only as good as its grader, and graders are attackable. The Reward Hacking Benchmark defines its threat model as “evaluation-mechanic exploitation: agents may discover and target weaknesses in the task setup, harness, parsing, or workflow constraints.” This is not hypothetical. The same paper documents an o3 run that listed its workspace, discovered a grader-internal file _meta/trace_index.json that was never referenced in the task spec, and copied the precomputed IDs into its output in two tool calls. The paper also notes this class of exposure “matches publicly available agent-evaluation harnesses including SWE-bench (Jimenez et al., 2023), in which test files live in a workspace that the agent can read.” If your checkers are exploitable, a high score may measure exploit-finding rather than work.

Reset and cleanup reliability. A long-horizon environment must return to a known state between runs, or your results are contaminated by the previous episode’s debris. This is a measured, imperfect quantity, not a property you get for free. HALTER, a purpose-built long-horizon evaluation harness for a Franka robot arm, restored the scene in only 76% of episodes, against 52% for AutoEval and 65% for a motion-planning reset baseline. HALTER also reports what evaluation costs a human: it cut operator interventions from 100 to 25 across 100 episodes and operator time from 88 to 24 minutes. Reset reliability is an axis you budget for, not a detail.

Run-to-run variance. A single-run score hides whether the agent is reliable or lucky. The stricter reporting convention appearing in this literature is Passˆ3: the fraction of tasks passed in all three independent rollouts, as reported in the WorkForge preprint’s description of Claw-Eval. This is materially stricter than Pass@3, which counts a task as passed if any one of three rollouts succeeds. The two numbers can diverge enormously for an inconsistent agent, and vendor and paper reporting do not always make clear which one you are reading. For a rollout gate, consistency is the quantity you actually care about; a team shipping an agent into a workflow cares whether it passes every time, not whether it passed once.

Path 1: the off-the-shelf benchmark, used honestly

Public benchmarks are not useless. They buy you comparability: the ability to say “this model scores X on the same tasks other teams measured,” which is genuinely valuable for vendor selection, regression tracking across model versions, and internal communication. What they do not buy you is evidence about your workflow.

If you use one, two disciplines matter.

Pin the release. Benchmark results drift as task data, harness code, and environment images change underneath them. The OSWorld 2.0 paper addresses this directly by defining comparable releases through compact manifests that specify the task dataset tag, website code tag, OSWorld code tag, a task hash manifest, and provider-specific Ubuntu images; its own experiments pin release v2026.06.24. If you cite a benchmark score internally without recording which release produced it, you will not be able to reproduce your own evaluation next quarter, let alone compare it to another team’s.

Mind the subset. Published numbers are often computed on subsets, and subset choices matter. The WorkForge preprint, for instance, reports its model comparison on a 201-task subset of GDPval’s official 220-task evaluation set, excluding 19 multimodal tasks. That is a reasonable engineering choice, but it means the number is not directly comparable to results on the full set. Before you compare two scores, confirm they were computed on the same tasks.

The honest summary of this path: a pinned public long-horizon benchmark gives you a coarse, comparable, generic signal. For a yes/no rollout gate on a specific workflow, coarse and generic is the wrong shape of evidence.

Path 2: the custom verifiable environment

The alternative is a small environment that mirrors your actual workflow, built around three components the current literature describes concretely.

A sandboxed, uniform tool surface. The HANDBOOK benchmark offers a clean template: 65 agentic tasks modeled on enterprise employees following company handbooks, each running in a self-contained Docker container with fixed resources (2 CPUs, 4 GB memory) from a common base image, exposing a uniform surface of 82 tools across six MCP servers reverse-proxied through a single streamable-HTTP endpoint. The design properties to copy are not the specific numbers; they are per-task isolation, fixed resources, a uniform tool surface, and mock services standing in for production systems. Those properties are what make results attributable to the agent rather than to environment accidents.

Programmatic checkers, audited as attack surface. The appealing property of a verifiable environment is that success is checked by code, not vibes: in WorkForge’s framing, “programmatic checks verify deterministic requirements, while semantic rubrics assess open-ended deliverables against the same facts”. But the Reward Hacking Benchmark’s o3 episode shows the checker and its scaffolding are part of the agent’s observable environment. The practical consequence: audit your harness the way you would audit any exposed surface. Grader-internal files, traces, and precomputed artifacts must not be readable from the agent’s workspace, and reads of anything like _meta/** should be logged as leakage events, as the RHB integrity monitor does. Note also that checker quality varies even among serious harnesses: HALTER’s verifier reached 91.0% accuracy against AutoEval’s 78.0% across 100 episodes. Your checker has an error rate, and until you measure it, a “pass” has an unknown false-pass probability folded into it.

Measured reset and multi-rollout reporting. Borrow HALTER’s move and treat scene restoration as a first-class metric with a number attached, not an assumption. If your environment only resets cleanly some of the time, some fraction of your “failures” are contamination and some of your “passes” rode on leftover state. Then report results across multiple independent rollouts, Passˆ3-style, so the gate measures consistency rather than luck.

The cost here is real and worth naming plainly: this path shifts the evaluation bottleneck from running tasks to writing trustworthy checkers and maintaining a resettable environment. That is engineering time your team spends instead of reading a leaderboard. The argument for spending it is that the leaderboard was never measuring your workflow in the first place.

The failure checklist

Whether you adopt a benchmark, build your own environment, or (most likely) do both, these are the four failure modes the documented evidence says to check before trusting a number:

1. Gamed checkers. Can the agent read anything the grader uses: test files, trace indices, precomputed artifacts? The RHB’s o3 episode passed a checker in two tool calls this way. Audit for it.
2. Stale snapshots. Do you know exactly which release of the benchmark or environment produced your score, down to task hashes and OS images? OSWorld 2.0’s manifest mechanism exists because unpinned results drift.
3. Unmeasured cleanup. What fraction of episodes does your environment actually restore to a known state? If you cannot answer, you have an unmeasured confound; HALTER measured 76% in a system built specifically to do better.
4. Single-run variance. Is the reported score one rollout, any-of-three (Pass@3), or all-of-three (Passˆ3)? For a rollout gate, only the last one measures what you care about.

What the WorkForge preprint does and doesn’t establish

Because it is the news peg, the WorkForge preprint deserves precise treatment. What it establishes, subject to the caveat that everything in it is author-reported and unreplicated as of 2026-10-10: verifiable environments with real artifacts, rich file types, and cross-file dependencies can be constructed at a scale of 16.7K environments across 40 professional domains, and the paper’s model comparison reports the stricter Passˆ3 consistency metric described above.

What it does not establish: that off-the-shelf verifiable environments are ready to serve as your deployment evaluation. The headline gains in the preprint, fine-tuning Qwen3.5-35B-A3B-Base on WorkForge-generated trajectories and improving five benchmarks including gains of 28.1 points on GDPval and 16.3 points on APEX, come from a training pipeline. Generating fine-tuning trajectories is a different use from gating a production rollout, and evidence for the first does not transfer to the second without separate validation. Its GDPval comparison also used a 201-task subset excluding 19 multimodal tasks, as noted earlier. The scale claim, if it replicates, suggests generic verifiable-environment coverage may grow quickly and is worth revisiting. It does not yet change the build-versus-use decision for a team shipping an agent this quarter.

Verdict, and where this advice may not generalize

My recommendation, with its conditions: demote public benchmarks to what they are good at, coarse comparability on pinned releases, and put your rollout gate on a small custom environment that reproduces your workflow’s task horizon, with audited checkers, an isolated and uniform tool surface, measured reset reliability, and multi-rollout consistency reporting. I would rather gate a rollout on, say, thirty tasks I trust than three thousand I have not audited, because every documented evaluation failure mode lives in checker and environment quality, not in task volume.

The limits deserve equal weight. No study in the evidence base directly compares outcomes of public-benchmark-guided versus custom-environment-guided rollout decisions; this framework is a synthesis of documented failure modes, not a controlled trial of the two paths. The reset and verification numbers come from robotic manipulation on a Franka arm and the long-horizon collapse findings from simulated retail; whether those rates transfer to software-work agents is unverified. And the WorkForge numbers that make this piece timely are unreplicated. If your tasks are short, self-contained, and well covered by an existing pinned benchmark, the custom path may not pay for itself. If a failed agent run in your environment means rebuilding a customer’s work, the cost of building trustworthy checkers is the smaller number.

Frequently Asked Questions

What is the difference between Pass@3 and Pass^3 in agent evaluation?

The stricter reporting convention appearing in this literature is Passˆ3: the fraction of tasks passed in all three independent rollouts, as reported in the WorkForge preprint’s description of Claw-Eval. This is materially stricter than Pass@3, which counts a task as passed if any one of three rollouts succeeds. The two numbers can diverge enormously for an inconsistent agent, and vendor and paper reporting do not always make clear which one you are reading.

How reliable is environment reset in long-horizon evaluations?

HALTER, a purpose-built long-horizon evaluation harness for a Franka robot arm, restored the scene in only 76% of episodes, against 52% for AutoEval and 65% for a motion-planning reset baseline. HALTER also reports what evaluation costs a human: it cut operator interventions from 100 to 25 across 100 episodes and operator time from 88 to 24 minutes. Reset reliability is an axis you budget for, not a detail.