Sixteen points of held-out benchmark success, without touching the model: that is the author-reported headline from Scale and Selection, a September 2026 arXiv study in which an automated loop edited only an agent’s harness and moved held-out success on the LIBERO robot-manipulation benchmark from 51% to 67% over 30 rounds (paper). The task agent was deliberately left alone, an unmodified Codex coding agent driving a simulated arm through screenshots and tool calls, so every point of the delta is attributable to scaffold changes rather than model improvements. If that result holds up outside its authors’ runs, it changes the budgeting question for small teams building GUI and computer-use agents: before you pay for a bigger model, you may be able to buy points with a disciplined harness-improvement loop.
The catch is embedded in the same result. The loop only worked under a strict acceptance rule backed by enough rollout evidence: the same strict rule run on small batches produced no reliable gain, and a permissive rule made the harness worse. This article is about where the 16 points came from, what conditions produced them, what the evidence cannot settle, and how to decide between three options your team actually faces: automate harness evolution, tune the harness by hand, or wait for the next model release.
What changed, and what deliberately didn’t
The experimental design is the reason to take the paper seriously. According to the authors, “the task agent is an unmodified coding agent (Codex with GPT-5.6 Sol) that controls a simulated manipulator by taking screenshots of a browser-based 3D interface and calling a small set of MCP tools to pose a virtual target gripper, execute waypoints, and end the episode.” The optimizer agent proposed edits to that surrounding scaffold: the observation formatting, the tool wrappers, the interface the task agent actually perceives and acts through. The model weights stayed frozen, and the task agent itself never changed.
That isolation matters more than the numbers. Anyone who has watched a computer-use agent’s score swing after a scaffold refactor, a change to how screenshots get annotated, a renamed tool parameter, a different system-prompt block, has suspected the harness carries a large share of measured performance. This study puts a number on the suspicion in one setting: with the agent frozen, 16 held-out points appeared purely from iterated scaffold edits. Combined with a pretrained policy, handed control through a single tool, the system reached “71.2% on the RoboCasa365 leaderboard benchmark, 13.1 points above the best published entry,” again author-reported (paper).
Two cautions apply immediately, and I will return to both. First, every figure here comes from the paper’s own runs, in a literature whose credibility problem the authors themselves quote: reported gains from harness evolution “often disappear under matched budgets and held-out tasks,” and “most claimed improvements on LIBERO are not statistically significant at the standard protocol” (related work). Second, the “+13.1” margin is measured against the authors’ snapshot of the best published RoboCasa365 entry, and leaderboard snapshots decay as new submissions land. Treat the numbers as evidence that harness effects can be large, not as settled scores.
Scale and selection are both load-bearing
The most actionable finding is not that an optimizer agent can improve a harness. It is that two conditions, both under the practitioner’s control, decided whether the same loop improved or regressed: how many rollouts stood behind each accept-or-reject decision, and whether a revision had to beat the incumbent to survive.
The paper lists scale first: “the number of rollouts the optimizer agent sees per round governs whether the evolved harness generalizes: with rounds fixed, held-out success rises as the training batch grows from 5 to 100 rollouts, while small batches overfit the training tasks” (paper). Run the same Champion, Challenger rule for the same 30 rounds with 5, 10, 20, and 100 rollouts per round, and the final champions scored 47%, 54%, 51%, and 67% held-out, against 51% for the baseline harness (Table 1). Only the 100-rollout campaign produced a clear gain; at five rollouts per round, the strictly gated loop finished four points below the baseline. The ten-rollout campaign is the sharpest warning: its champion hit 70% on its training cases, the highest training score of any campaign, and only 54% held-out (Table 1), having fitted the particular cases it saw. With so few cases, most challengers tie the champion exactly and are rejected, so small-batch loops also stall; and with five rollouts per round, one flipped outcome moves the success rate by twenty points, so a revision can be promoted purely by luck.
The second condition is the acceptance rule. The paper reports that “accepting every revision it proposes degrades the harness within a few rounds; a simple Champion–Challenger rule that promotes a revision only if it beats the incumbent on the same fixed cases turns the same loop into one that raises held-out success on LIBERO tasks from 51% to 67% over 30 rounds” (abstract).
Read that twice if you are planning to build this. The naive version of automated harness improvement, let an agent propose scaffold edits and take them, actively destroys the artifact. The working version is mundane engineering discipline: a frozen set of evaluation cases, an incumbent champion, and a challenger that must beat the champion on those exact cases to be promoted. Nothing about the rule is exotic. What the study contributes is evidence on both failure modes: without the gate, the loop regresses; with the gate but on thin batches, it overfits or stalls.
The practical consequence: the fixed case set is the asset, and its size is part of the asset. If you adopt this pattern, your eval cases need the same protection as your model weights would get in a fine-tuning pipeline: versioned, frozen during a run, and refreshed only deliberately. Each promotion decision also needs enough cases behind it that a couple of lucky outcomes cannot carry a bad revision, which in this paper meant batches of about 100 rollouts per round (Table 1), with every round scoring both the champion and the challenger on them. That batch size is also what sets the campaign’s cost. A loop that drifts its own evaluation set while evolving the harness would silently reward scaffold overfitting, and the paper’s accept-all baseline is the warning label for what unchecked optimization does.
Anatomy of the evolved harness
It is worth being concrete about what “the harness” meant here, because the components are recognizable to anyone building computer-use agents. The paper defines it as everything between the model and the robot, and the surfaces the optimizer’s gated edits actually touched fall into familiar layers:
- Prompts and operating guidance: the system prompt and the textual operating guide the task agent reads.
- Observation formatting: screenshots of a browser-based 3D interface and the observation returned after each tool call, rather than a policy-native action space or a structured scene description.
- Tool set and control rules: a small set of MCP tools exposing discrete capabilities (pose a virtual gripper, execute waypoints, end the episode), the semantics of each tool, and the control, verification, and termination logic around them.
That anatomy is why the result is interesting beyond robotics. A GUI agent stack typically has the same surfaces: what the model is told, what it sees, and what actions it can invoke. The paper’s claim, at least within its setting, is that these are optimizable surfaces, and that an agent proposing edits to them, gated by Champion, Challenger selection, finds real gains.
One further component is easy to misread as part of that evolved harness. The handoff tool that passes control to a pretrained policy was not a surface the loop optimized: the authors added it manually after the campaign finished, writing that “Starting from the round-2 champion, we added one MCP tool, vla_run,” to combine the evolved harness with the policy for the RoboCasa365 leaderboard run (paper). The 16 held-out points came from the gated edits alone; the 71.2% leaderboard score came from the edits plus this deliberate author addition. Keep the two apart when you price the pattern.
Does it transfer? Robot sims vs OSWorld vs coding agents
Here the evidence thins. The 16-point result and the 71.2% leaderboard score (paper) are robot-simulation numbers, but harness evolution itself is not confined to simulation. The paper’s related-work section notes that “harness changes alone shift held-out success by 10 to 15 points on tool-use benchmarks” and that “an automatically discovered harness can outrank hand-built agents on Terminal-Bench” (Scale and Selection). UI-Evol is automated harness-side knowledge evolution measured on OSWorld: it lifted Agent S2 from 19.5% to 22.0% success under GPT-4o and from 25.6% to 28.4% under OpenAI-o3 (Table 1), author-reported. None of these adjacent numbers escapes the matched-budget critique quoted above, and if your product is a desktop automation agent or a coding agent you are still extrapolating from neighboring settings, so size your investment accordingly.
Two adjacent findings bound the extrapolation from opposite directions.
On the pessimistic side, UI-Evol, which studied computer-use agents on OSWorld, reports that “even 90% correct knowledge yields only 41% success rate” (paper). Correct harness-side knowledge, accurate guidance about how to operate the interface, caps out well below task success. Execution, grounding, and recovery behavior carry the rest. This cuts against a naive reading of Scale and Selection: polishing what the scaffold tells the agent is not sufficient on harder, real-desktop tasks, and a harness loop tuned to knowledge correctness could plateau far from your target.
On the more encouraging side, the UI-Evol work also documents the evaluation infrastructure that any harness loop needs, and it is cheaper than you might fear: “30 instances, which reduces the full-process runtime of the OSWorld benchmark from approximately 10 hours to 2.5 hours under maximum 15 steps” (paper). A Champion, Challenger gate is only as good as the evals behind it, and 2.5 hours per full OSWorld pass on parallel infrastructure is a workable inner loop for a small team. Not free, but not a frontier-lab budget either.
The honest summary: harness effects are large in robot sims, present but contested on tool-use and Terminal-Bench-style software benchmarks, and knowledge-side evolution on OSWorld lifts scores while leaving the execution gap UI-Evol documents. If you run an OpenHands-style coding agent, no result cited here measures your stack directly; whether harness evolution helps there is a question your own frozen-case experiment has to answer.
An interface-first harness, and its fine-tuning mode
The most useful adjacent paper is Show-Harness, and reading it as pure counter-evidence gets it wrong: it is itself a harness paper. Its contribution is an interface, an “Embodied Harness” that “exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions,” wrapping the same VLM into planner and controller roles inside the loop. Its argument is that the right interface draws substantial embodied capability out of foundation VLMs “without requiring additional model capacity,” demonstrated in two modes: closed-source frontier VLMs driving robots zero-shot with no fine-tuning, and lightweight open-source models adapted into the same action space, with both modes reported to outperform representative baselines.
That second mode is the model-side lever, and it is cheap. The authors report that their approach “enables lightweight VLMs (e.g., 2B-scale models) to ‘play’ robots in the same action space with just a few GPU-hours of fine-tuning, achieving stronger generalization and sim-to-real transfer than representative VLA paradigms under controlled experiments” (Show-Harness).
That is a different bet with a different profile. Harness evolution costs evaluation compute and optimizer-agent calls, keeps you model-agnostic in principle, and needs no training infrastructure. Fine-tuning a 2B-scale VLM costs a few GPU-hours per iteration, needs training competence and data, and buys you a policy that (in this paper’s controlled experiments) generalized better and transferred to real robots. The two papers also price their levers in different units: Show-Harness counts GPU-hours of fine-tuning per iteration, while Scale and Selection counts roughly 100 rollouts per round for 30 rounds (paper); a head-to-head budget comparison would have to come from your own measurement. It is also single-study and author-reported, so this is not a verdict. It is evidence that the model-side lever is cheap enough, at small scale, that “harness first” cannot be a default. It has to be a decision.
At the heavyweight end, MagicGUI shows what the model-side path looks like with real resources: continue pre-training on 7.8M samples followed by reinforcement fine-tuning with a spatially enhanced composite reward. That path is not available to most teams, which is precisely why the harness question matters, but it sets the reference point for what you are choosing not to do.
Two costs the headline hides
Model, harness coupling. An evolved harness is tuned, implicitly, to the model that ran inside it, and the evidence on how much that matters is mixed. ENPIRE makes the coupling risk concrete with a real-world manipulation comparison: “Codex solves the task fastest, while the Codex harness using the Opus API is the least efficient among the tested configurations.” The same harness, different model API, flipped efficiency ranking. Scale and Selection’s own cross-model test cuts the other way: re-running its baseline harness and evolved champion with GPT-6 Astra, “a newer and stronger model that the optimizer never saw,” the authors report that “the harness gain is not absorbed by the stronger model: under Astra it is worth 46 points, more than under Sol” (Table 3). One real-robot study finds harness quality model-relative; one robot-sim study watched its gains survive and grow under a model it never saw. For planning purposes, treat a model swap after harness optimization as a re-validation event, not as a known loss or a known free ride.
Maintainability. A scaffold that accumulates 30 rounds of automated edits is a scaffold your team may no longer be able to read. The Harness Handbook paper exists because this is a real failure mode; it proposes Behavior-Guided Progressive Disclosure (BGPD), which “guides coding agents from high-level behavior descriptions to relevant implementation details in stages,” evaluated on two open-source harnesses. If an evolved harness becomes opaque, you inherit an artifact that scores well and that nobody can safely modify by hand. I would treat readability tooling, or at minimum a convention of documented harness structure, as part of the loop’s cost from day one, not as cleanup.
The manual middle option
Between full automation and waiting sits the low-tech lever: hand-written, reusable scaffold knowledge. Work2Skill, posted in October 2026, evaluates induced skills injected as prompt guidance, “appending their text as a clearly delimited block to the end of the executor’s base system prompt,” with a shared <task_skills> wrapper for portability across execution models. No optimizer agent, no acceptance loop, no eval infrastructure beyond what you already run.
This is the right starting point for a team that has not yet frozen an evaluation set. Manual skill injection forces you to articulate what the harness should know, and the delimited-block pattern keeps it swappable. The limitation is the UI-Evol ceiling: knowledge correctness alone does not close the execution gap, and a manual process will not explore the scaffold space the way 30 rounds of gated proposals did.
Decision guide
| Option | Upfront cost | Evidence for gains | Main risk | Best fit |
|---|---|---|---|---|
| Automated harness evolution (Champion–Challenger on frozen cases) | Eval infrastructure and compute for roughly 100 rollout cases per round (champion and challenger each scored on them), optimizer-agent calls, fixed case set | Author-reported +16 held-out points (LIBERO), robot sims only | Small-batch overfitting; accept-all regression; unreadable scaffold; re-validation after a model swap | Teams with a stable eval set and a frozen model |
| Manual harness tuning (skill injection, tool-wrapper edits) | Engineer time | Work2Skill pattern; gains bounded by the execution gap (UI-Evol: 90% correct knowledge, 41% success) | Execution-gap ceiling; slow exploration | Teams without a frozen eval set yet |
| Fine-tune a small model | A few GPU-hours per iteration, training competence | Author-reported generalization and sim-to-real gains (Show-Harness fine-tuning mode, controlled experiments) | Training skill, data pipeline, less model-agnostic | Teams with ML capacity and a fixed action space |
| Wait for the next model release | Zero now | Mixed on whether gains survive a swap: worth 46 points under an unseen model in one test (Table 3); harness rankings flipped across model APIs in ENPIRE | Falling behind while waiting; re-validation cost later | Teams whose bottleneck is not the harness |
My Read of this evidence: if you do not have a frozen, versioned evaluation set, build one first regardless of which row you pick, because three of the four rows are unmeasurable without it. If you have the eval set and no frontier-model budget, a Champion, Challenger harness loop is the cheapest credible next spend, and the paper’s own accept-all baseline tells you the gate is non-negotiable. Budget the batch size with the same seriousness: the clear gain arrived at 100 rollouts per round (Table 1), while the same gated loop at smaller batches never clearly beat its baseline, finishing at 47%, 54%, and 51% held-out against 51% (Table 1). If you have training competence and a stable action space, run the Show-Harness-style fine-tuning experiment in parallel rather than choosing blind; a few GPU-hours is cheap enough to be a measurement, not a commitment.
Hold two limits in view. Every quantified harness gain cited here is author-reported, and the largest numbers come from one study of simulated manipulation, in a literature that the study itself says is prone to gains that vanish under matched budgets. And the leaderboard framing will age: the 71.2% and +13.1 figures describe one snapshot of RoboCasa365 (paper). What survives is the structure of the finding: the harness carries a large, measurable share of the score, rollout scale and a strict acceptance rule together decide whether automation helps or harms, and the one direct test found the gains transferring to a model the optimizer never saw. That is enough to act on, carefully, and not enough to skip your own frozen-case experiment.
Frequently Asked Questions
How many rollouts per round are needed for the automated harness evolution loop to produce a clear gain?
Only the 100-rollout campaign produced a clear gain; at five rollouts per round, the strictly gated loop finished four points below the baseline. The ten-rollout campaign is the sharpest warning: its champion hit 70% on its training cases, the highest training score of any campaign, and only 54% held-out (Table 1), having fitted the particular cases it saw.
What acceptance rule prevents the automated harness evolution loop from degrading the harness?
The paper reports that “accepting every revision it proposes degrades the harness within a few rounds; a simple Champion–Challenger rule that promotes a revision only if it beats the incumbent on the same fixed cases turns the same loop into one that raises held-out success on LIBERO tasks from 51% to 67% over 30 rounds” (abstract).
Does the harness gain survive when the model is swapped for a newer, stronger one?
Scale and Selection’s own cross-model test cuts the other way: re-running its baseline harness and evolved champion with GPT-6 Astra, “a newer and stronger model that the optimizer never saw,” the authors report that “the harness gain is not absorbed by the stronger model: under Astra it is worth 46 points, more than under Sol” (Table 3).
Join the discussion
Share a useful perspective or ask a question about this article.