When an agent benchmark score jumps after a scaffold upgrade, the jump often belongs to the harness rather than the model, and a new preprint tries to quantify how often. Rethinking the Evaluation of Harness Evolution for Agents (arXiv 2607.12227, v2 posted 2026-08-27) is a single author-reported, non-peer-reviewed submission, tested on one benchmark family (Terminal-Bench 2.1) with two frontier models (GPT-5.4 and Claude Opus 4.6). Treat its numbers as unreplicated. Its protocol critique, though, stands on logic alone, and it changes how internal agent evals should be reported.
What does the preprint actually claim, and how far does it reach?
The paper claims that automatic harness evolution, the practice of searching over agent scaffold configurations and then reporting the best configuration’s score, does not consistently outperform simple test-time scaling once both methods receive the same feedback and inference budget, and that the harnesses it produces show limited generalization, including on held-out tasks the search never touched. That is a narrower and more damaging claim than “harnesses matter.” It says a whole category of published scaffold gains may be search gains wearing a harness costume.
The submission history matters for how much weight to put on it. Version 1 was submitted on 2026-07-14 by Yike Wang, and the v2 revision landed on 2026-08-27, two days before this writing. The authors call for fairer evaluation protocols and benchmarks for automatic harness design and state that their code is publicly available, which at least makes the experiments checkable in principle.
The provenance caveats are structural, not incidental. arXiv’s own about page states that material on the site is not peer-reviewed, that contents are wholly the submitter’s responsibility, and that everything is presented as is, without warranty. Wikipedia’s arXiv entry records a submission rate of roughly 24,000 articles per month as of November 2024, and notes that moderation checks topicality and scholarly value but is not peer review. In November 2025, arXiv stopped accepting computer-science review articles and position papers that had not already been vetted by a journal or conference, citing a rise in AI-generated research. None of that makes this paper wrong. It means the burden of checking sits with the reader, which is the normal state of affairs for preprints and worth stating plainly before anyone quotes its numbers at a planning meeting.
What is harness evolution, and why do its scores resist easy reading?
Harness evolution treats the scaffold around the model, the prompts, tool wiring, retry logic, context management, and control flow, as an object to be optimized by iterative search, which makes the final benchmark score a joint measurement of three things at once: the model, the harness, and the search procedure that produced the harness. The paper describes methods that search for harness configurations using unit test cases and then report final performance on the same public benchmark. The score at the end of that pipeline has no obvious decomposition.
This is the same problem internal teams face every time a Claude Code-style harness ships an update. The scaffold is a moving measurement instrument. A model that scored X under harness version A and Y under harness version B has not necessarily changed at all; the ruler changed. When the harness itself was found by an automated search loop, there is a third variable folded in: the compute and task feedback spent searching. Comparing that number against a model run under a fixed, hand-written harness is not a comparison of models, and the paper’s central objection is that the field has been reading it as one.
It helps to be precise about what “harness” covers here, because the term gets used loosely. In this paper’s frame it is everything in the agent loop that is not the weights: system prompts, tool schemas, how tool outputs get parsed and re-injected, retry and fallback behavior, subagent delegation, context-window budgeting. Every one of those is a design decision a team revises between releases, and every revision is a silent protocol change to any eval running on top of it.
Which two protocol flaws make the scores hard to attribute?
The paper identifies two specific flaws: harness search and final evaluation typically share the same benchmark, so reported gains risk overfitting to that exact task set, and harness evolution consumes extra feedback and inference budget during its search loop, so without a matched-budget baseline there is no way to separate harness-design gains from gains that come from extra search alone. Both flaws inflate the number. Neither requires any dishonesty on the part of the researchers; they are defaults that emerge when the search procedure and the scoreboard live in the same place.
The shared-benchmark problem is the older and better understood of the two. If a procedure repeatedly evaluates candidate harnesses against a task set and keeps the winners, the final harness is partially a memorization of that task set, even if no individual task appears verbatim in a prompt. The selection pressure itself leaks the benchmark into the artifact. The paper’s authors argue this directly: because the harness search and the final evaluation share the same benchmark, reported gains risk overfitting to that specific task set. Their held-out-task experiments are the empirical check, and the limited generalization they report is exactly what the overfitting hypothesis predicts.
The unmatched-budget problem is subtler and, for practitioners, more actionable. Automatic harness evolution is an iterative search procedure that evaluates and revises candidate harnesses using task feedback. Every one of those evaluations is inference. Every revision informed by task feedback is, functionally, a training signal. The paper’s argument is that a method consuming this much extra task-level interaction must be compared against simple task-level search baselines, things like best-of-n sampling or retry-until-pass strategies, given the same feedback and inference budget. If the evolved harness beats a fixed harness but ties a dumb search baseline at equal budget, the honest headline is “search helps,” not “harness design helps.”
What does this change for teams running internal agent evals?
For a team running Claude Code-style harnesses internally, the practical consequence is an attribution rule: when a score moves after a harness iteration, the default explanation is “the protocol changed,” not “the model got better,” until an ablation says otherwise. This inverts the common reading of internal dashboards, where a green delta after a harness release tends to be celebrated as a capability improvement regardless of which variable moved.
The inversion has teeth because scaffold sensitivity is real and measurable. A separate preprint from the same week, arXiv 2608.27146, shows scaffold-level design moving agent outcomes in a security context: its SARA runtime-authorization architecture holds attack success rate to no more than 0.63% across four primary settings on AgentDojo and AgentDyn while maintaining competitive task utility, according to its authors. Scaffold choices move numbers. That is precisely why an unattributed score delta is uninformative: the harness is a real causal variable, so you have to control for it, not ignore it.
The disclosure gap makes this the internal team’s problem rather than the vendor’s. Consider Microsoft’s Office Agent FAQ for its preview agent: evaluation is described in terms of Responsible AI review, harm filtering, and prompting users to fact-check outputs, with no benchmark scores and no harness or version information disclosed. That is a normal vendor document, not an outlier. If your eval pipeline wraps a managed agent product, the vendor will not tell you when the harness under your score changed. Pinning and reporting have to happen on your side of the API boundary, or they do not happen.
There is also a neighboring-literature signal that this is not one paper’s idiosyncrasy. arXiv 2608.23780, submitted 2026-08-24, argues in a different domain, LLM-based measurement of student talk, that the standard validation toolkit of expert-annotation comparison, held-out sets, and F1 scores is insufficient to establish that LLM-based measures are valid at all. One preprint per domain does not make a movement, but the same evaluation-validity concern surfacing independently in agent benchmarks and educational measurement suggests the underlying issue is general: the field’s default protocols for “did the number move for a real reason” are weaker than the decisions being made on top of them.
What should an eval report pin alongside every score?
Every agent eval score should ship with five pieces of metadata: the harness version, an ablation separating scaffold changes from model swaps, a matched-budget search baseline, held-out-task results, and an explicit statement of benchmark and model coverage. Without all five, a delta between two runs is an observation, not a measurement.
The checklist operationalizes the paper’s two objections plus the coverage caveats its own authors would presumably endorse:
| Reporting practice | What it isolates | What breaks without it |
|---|---|---|
| Harness version pinned beside the score | The measurement instrument itself | Scores from different scaffold versions get compared as if comparable |
| Scaffold changes and model swaps ablated separately | Which variable caused the delta | Attribution defaults to whichever story the team prefers |
| Matched-budget search baseline (e.g., best-of-n at equal feedback and inference) | Harness-design gains vs. plain extra search | Search spend masquerades as design quality |
| Held-out task results | Generalization beyond the tuning set | Overfitting to the benchmark the harness was selected on |
| Benchmark and model coverage stated | The claim’s blast radius | A two-model, one-benchmark result gets read as a law of nature |
None of these rows requires new research. They are reporting discipline. The matched-budget row deserves emphasis because it is the one most likely to be skipped: running a best-of-n or retry baseline at the same feedback and inference budget as your evolved or upgraded harness is cheap compared to the harness work itself, and it is the only control that answers the question a stakeholder actually asks, which is “was that worth it, or would more samples have done the same thing?”
A second operational habit follows from the shared-benchmark objection: maintain a held-out task set the harness iteration process never touches, and run it on any harness candidate before promoting it. The set does not need to be large or published; it needs to be firewalled from whatever loop, human or automated, is proposing scaffold changes. The paper’s limited-generalization finding on held-out tasks is the empirical argument that this firewall is load-bearing.
How far does the evidence stretch, and where does it stop?
The evidence supports a protocol critique and a default-attribution rule, and it does not support the conclusion that harness design is a waste of time. Two boundaries matter, and getting either wrong misreads the paper.
The first boundary is scope. Everything here rests on one author-reported, non-peer-reviewed arXiv submission, evaluated on a single benchmark family with two frontier models. More importantly, the studied object is automatic harness-evolution search procedures: methods that run an explicit optimization loop over harness configurations. Generalizing the result to every harness iteration a team ships, including hand-designed updates informed by error analysis, is an extrapolation the experiments do not directly cover. A human engineer reading failure traces and fixing a retry bug is not the same process as an automated search consuming thousands of task-feedback evaluations, and the paper’s matched-budget objection bites much harder on the latter than the former.
The second boundary runs the other direction. The headline result is that automatic harness evolution “does not consistently outperform” simple test-time scaling under matched budgets. “Does not consistently outperform” is not “never outperforms,” and the paper does not claim that harness design is irrelevant, only that its gains are easy to confuse with search gains and slow to generalize. The SARA result cited above is the reminder that scaffold architecture can be the entire ballgame in some dimensions; a 0.63% attack success rate, if it replicates, is not a rounding error on the model’s behavior. The correct takeaway is that scaffold gains are real but must be proven, per change, with controls. The incorrect takeaways are “harnesses don’t matter” and “my scaffold upgrade definitely made the model smarter.” The paper sits exactly between those two overreads.
The practical verdict: separate scaffold gains from model gains
The working rule is attribution by default. Pin the harness version next to every agent score, treat scaffold changes and model swaps as separate experiments, and require held-out-task results plus a matched-budget search baseline before crediting either variable. If a score moved after a Claude Code-style harness iteration, the default attribution is “protocol changed,” not “model got better,” until an ablation says otherwise. This costs a held-out task set, a best-of-n control run, and a version string in a table column. That is a cheap insurance premium against re-baselining your entire eval suite every time the scaffold ships.
The strongest limitation deserves the last word, because it bounds how much of this you should act on. The case rests on a single author-reported, non-peer-reviewed preprint, evaluated on one benchmark family with two frontier models, and its direct subject is automatic harness-evolution search rather than the full space of harness iteration. The specific empirical claims, no consistent advantage over test-time scaling, limited held-out generalization, could weaken or flip on replication across more benchmarks and models. The protocol recommendations do not depend on those specific numbers surviving, because matched budgets, held-out tasks, and version pinning are sound measurement hygiene regardless of who is right about harness evolution. That asymmetry is the reason to adopt the checklist now and the headline numbers only after someone replicates them.
Frequently Asked Questions
Does the matched-budget baseline requirement apply to hand-tuned scaffold updates?
The paper’s matched-budget objection targets automated search loops that consume thousands of task-feedback evaluations. For hand-tuned updates based on error analysis, the compute cost is negligible, so the primary control needed is a held-out task set to check for overfitting, rather than a best-of-n search baseline.
How does the SARA architecture result contrast with the harness evolution findings?
While the harness evolution paper finds limited generalization for searched scaffolds, arXiv 2608.27146 shows that a specific, hand-designed runtime-authorization architecture (SARA) holds attack success rates to 0.63% on AgentDojo and AgentDyn. This demonstrates that deliberate scaffold design can yield security gains that automated search fails to replicate or generalize.
What is the operational cost of implementing the five-point reporting checklist?
The primary cost is maintaining a firewalled held-out task set that is excluded from all scaffold iteration loops. This requires versioning the task set separately from the training or tuning data, ensuring that no human or automated process proposing scaffold changes has access to the held-out tasks during the design phase.
Why is the November 2025 arXiv policy change relevant to this preprint’s credibility?
arXiv stopped accepting unvetted computer-science review articles and position papers in November 2025 due to a rise in AI-generated research. This policy shift highlights the broader pressure on preprint credibility, reinforcing the need for independent replication of author-reported results like those in arXiv 2607.12227 before treating them as established fact.
