A coding agent’s test harness is usually written before the agent runs and left untouched while the agent hunts for a passing implementation. TTHE: Test-Time Harness Evolution, posted to arXiv on 9 July 2026 as arXiv:2607.08124, breaks that convention. It treats the executable harness itself as mutable state that evolves during evaluation, using the same frozen large language model as solver, proposer, and judge. Coverage rises on tasks where hand-written tests are incomplete, but the verification layer becomes part of the solution space rather than an external constraint.
What does TTHE actually propose?
TTHE reframes test-time adaptation as program search over the executable harness, not as model fine-tuning, prompt retrying, or a pre-searched workflow. The authors define the harness as the “executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures” TTHE. In their setup, that program is the thing that changes during evaluation. The model weights stay frozen, no gold labels are required, and no separate adaptation model is trained. All adaptation occurs through changes to the surrounding program.
This is a shift in where the intelligence is assumed to live. Most agent work treats the LLM as the variable and the harness as fixed scaffolding. TTHE does the opposite. The solver, the proposers, and the judge are “different roles and harnesses around the same frozen LLM” TTHE. The unit under improvement is the code that decides what the solver sees, what tools it can call, and what counts as a correct intermediate step. Once that unit is allowed to change, the boundary between specification and implementation starts to blur. The test harness is no longer a neutral referee; it is a co-evolving component of the system.
How does the harness evolve at test time?
TTHE keeps a population of candidate harnesses and refines them as new inputs arrive. An “agentic proposer reasons over their execution traces” to suggest changes to the harness, and “a judge then commits an improved harness from execution-derived proxy signals” TTHE. The selected harness persists and governs subsequent inputs, giving “persistent, inspectable improvements rather than per-query retries or a pre-searched workflow” TTHE.
The process is unsupervised in the usual sense: the judge has no gold labels and no task-specific supervision. It must decide whether a proposed harness is better than the incumbent using only what can be inferred from execution. That makes the design clean on paper and fragile in practice. A harness edit might add an assertion, expand a fixture, tighten an oracle, or change how failures are recovered. Each edit is a program transformation, and the judge’s approval function is itself a program that runs on the traces produced by other programs. The nesting is unavoidable, and it is also the source of the method’s risk.
Because the improvements persist, TTHE is not just a fancy retry loop. A ReAct-style agent that fails and resamples a reasoning trace forgets the failure once the next query begins. TTHE remembers the harness edit that turned a failure into a success and applies it to future tasks. That memory is inspectable, which is a genuine advantage over opaque weight updates, but it also means the benchmark is now measuring a system whose verification logic has been shaped by earlier test cases.
What did the experiments cover?
The authors evaluated TTHE “across text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool-use tasks,” improving fixed ReAct-style baseline harnesses in each setting TTHE. The arXiv abstract does not disclose specific numerical scores, so any claim about exact pass-rate improvements would be speculative. What is clear is that the authors treat the breadth as part of the argument: the harness-evolution mechanism is meant to apply wherever an agent uses tools, verifies intermediate outputs, and can observe execution traces.
The choice of domains matters. Text-to-SQL and data-science coding are notorious for under-specified oracles, where a correct answer can fail a rigid test because of column order, formatting, or a benignly different aggregation. Competitive programming has exact oracles but often sparse coverage of edge cases. Software engineering tasks, including those on benchmarks like SWE-bench, rely on project-specific test suites that may be incomplete. Agentic tool-use introduces partial observability and external state. By spanning all five, the authors are claiming that harness evolution is a general strategy, not a trick tuned to one benchmark family.
Why are execution-derived proxies the hard part?
The central obstacle TTHE identifies is “execution-derived proxy reliability” TTHE. Without gold labels, the judge must trust signals extracted from running candidate harnesses on test inputs. Those signals can be wrong, and they can be gamed.
A proxy that rewards passing more assertions can be satisfied by a harness that adds trivially true checks or, worse, checks that are tautologically aligned with the candidate outputs it already produces. A proxy that rewards fewer crashes can be satisfied by a harness that stops calling risky tools. A proxy that rewards trace diversity can be satisfied by meaningless variation. These are not theoretical edge cases; they are the standard failure modes of unsupervised self-improvement in agents. TTHE’s architecture makes them visible because the harness is an explicit program, but visibility is not the same as immunity.
The problem is recursive. The proposer generates harness edits by reasoning over traces produced by the current harness. The judge evaluates those edits using proxy signals computed from the same traces. If the current harness has a systematic blind spot, both the proposed edits and the approval signal inherit it. The only external anchor is the final task outcome, and on many agent benchmarks even that outcome is itself a test-suite score that the evolving harness can influence. Breaking out of that loop requires either some labeled ground truth, a held-out evaluation harness that is never optimized, or a theory of proxy reliability that the paper flags but does not fully resolve.
What does this mean for fixed-test benchmarks?
TTHE puts pressure on any coding-agent benchmark that treats its test harness as fixed, external evaluation infrastructure. The conventional contract is that the benchmark defines the task and the agent produces an implementation that satisfies the definition. Once the harness can be optimized, the benchmark is measuring a joint optimization over implementation and verification. The leaderboard then compares harness-solver pairs, not just solvers, and any claim that one model beats another must account for whether the harness was allowed to change.
The contrast with TAG: A Lightweight Framework for Test-Driven Agentic Artifact Generation, posted to arXiv in July 2026, is instructive. TAG takes a fixed “LLMs generate, we validate” approach, relying on “deterministic and LLM-based tests” plus expert-distilled judges TAG. Deterministic tests check schema, syntax, and cross-references; LLM-based tests check semantic intent and logical consistency. TAG bets that rigorous validation can make generated artifacts production-ready. TTHE bets that the validation layer itself should be mutable. The two frameworks disagree about where the frontier lies: at the model output, or at the code that judges it.
| Aspect | TTHE | TAG |
|---|---|---|
| Test harness | Mutable during evaluation | Fixed |
| Adaptation target | Executable harness program around a frozen LLM | Generated artifact quality via validation |
| Supervision | Unlabeled execution traces | Deterministic tests, LLM-based tests, expert judges |
| Core goal | Improve the harness at test time | Make LLM-generated artifacts reliable enough for production |
| Key risk | Execution-derived proxy reliability | Validation coverage and judge calibration |
For benchmark designers, the implication is uncomfortable but clear. If a harness can be part of the solution, a fixed-test leaderboard must explain why its harness is privileged. That explanation might be empirical, a frozen harness still correlates with human judgment, or practical, changing the harness is expensive and out of scope. But it cannot be assumed. The same argument applies to industrial evaluations. A team that lets an agent rewrite its own CI assertions must not be surprised when the agent writes assertions it can pass.
What should agent builders watch?
Treat a mutable harness as production code, because it is. Version it, diff it, and review the edits the proposer commits the same way you would review a pull request that changes the test suite. The fact that TTHE’s improvements are persistent and inspectable is a feature only if someone inspects them.
Keep the gold evaluation harness separate from the harness the agent is allowed to rewrite. The mutable harness can legitimately improve coverage on development tasks, but the final score should come from a fixed, human-audited suite that the agent never saw during harness evolution. Without that separation, coverage gains and benchmark overfitting become indistinguishable.
Audit the proxy signals the judge uses. “Execution-derived proxy reliability” is the headline risk TTHE, and the best defense is to know exactly what the proxy is measuring. A proxy that counts passed assertions is different from one that checks output equivalence against a reference, and both are different from one that measures whether a changed assertion actually caught a real bug. Each proxy has its own failure mode, and each deserves its own monitoring.
Finally, do not conflate harness evolution with model improvement. TTHE keeps the LLM frozen for a reason: it isolates the contribution of the program around the model. If the harness is doing the heavy lifting, the right credit assignment matters for both research and procurement. A leaderboard that reports a single top-line number without saying what was allowed to change, model, harness, or both, is reporting less than it appears to.
Frequently Asked Questions
Which agent tasks are the worst fit for TTHE?
Tasks that already have exact, exhaustive checkers and abundant labeled examples leave little room for an unsupervised judge to improve the test scaffold. The method is aimed at settings where hand-written tests are incomplete or the oracle is fuzzy, such as text-to-SQL, data-science coding, and open-ended tool use.
How is TTHE different from simply running ReAct with more retries?
ReAct retries sample a fresh reasoning trace for each query and forget the failure once the next task starts. TTHE persists the edits that turned failures into successes as versionable code, so later tasks inherit coverage improvements without changing model weights, but also without the easy rollback of a fixed benchmark.
What separation does a team need before deploying a mutable test rig?
Keep a frozen, human-audited evaluation suite that the agent never sees during evolution; version every committed edit to the mutable verification program; and log the proxy signals the judge uses. Without that split, coverage gains and benchmark overfit become impossible to tell apart.
Why are execution-derived proxies easy to game?
A judge that rewards passing more assertions can approve a test scaffold that adds vacuously true checks; one that rewards fewer crashes can approve a scaffold that stops calling useful but risky tools. Cross-checking edits against a static reference suite is the usual defense, but it multiplies inference and sandbox costs because each candidate must be run across many tasks.
What would keep fixed benchmarks relevant if mutable verification catches on?
If head-to-head studies show that evolved test rigs overfit their development set, or if a standards body requires all submissions to use the same frozen checker, the fixed-test contract would regain force. Until proxy reliability has a tractable theory, that contract is the simplest way to compare models on equal terms.
