Why Self-Improvement Claims Need Benchmarks the Agent Has Never Seen

All 35 model-game cells with valid self-test outputs in a study of heuristic self-improving agents ended with self-scores of at least 0.70, and 15 of those completed policies still scored below their game’s random baseline (Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents). When a vendor or internal team claims an agent “self-improved” through an automated loop, the score that backs the claim often comes from the same benchmark the agent iterated on. That score is cheap to manufacture and proves little. A newer proposal, learning to predict a modification’s effect before applying it, changes the economics of checking every change, but not the underlying question: a predictor trained inside the loop must itself be proven free of the contamination it is asked to police. Before a self-improvement claim crosses a deployment gate or lands in capability-change documentation, reviewers should demand evaluation on held-out tasks the agent never saw, benchmark lineage logs, and calibrated judges, and should treat any learned predictor of improvement as one more judge to calibrate. Otherwise they are not reviewing capability evidence; they are accepting the improvement loop’s own marketing.

What a self-improvement claim actually asserts

A self-improvement claim sounds simple: the agent changed, and the change made it better. Unpack the assertion and it contains three separate propositions a reviewer is implicitly signing off on. First, that a modification was promoted into the agent (a new prompt, policy, tool configuration, or fine-tune). Second, that the promotion was justified by a measurement. Third, that the measurement reflects capability on tasks the agent will actually face, not just familiarity with the tasks it practiced.

The third proposition is where claims fail. A survey of self-improvement in agentic systems, Self-Improvements in Modern Agentic Systems, states the requirement plainly: to confirm genuine capability gains rather than memorization of feedback, “evaluation should measure performance on a held-out distribution that does not overlap with the optimization data.” If the improvement loop and the evaluation draw from the same task pool, the score cannot distinguish an agent that got better from an agent that memorized its own test.

That distinction is the reviewer’s entire job. A deployment gate that accepts in-loop scores is not gating anything. It is certifying a number whose generating process included the answer key.

Self-authored verification manufactures improvement

The strongest empirical evidence comes from a study of heuristic self-improving agents, Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents. The setup let agents write their own tests and score themselves against them while improving policies for game-playing tasks. The author-reported result is stark: among the 35 model, game cells with valid self-test outputs, every single one ended with a self-score of at least 0.70. Yet 15 of those 35 completed policies scored below their game’s random reference baseline, including six policies sitting at Pong’s −21.0 floor, the worst possible outcome.

Read that as a reviewer would. Every self-improvement run reported success. Nearly half produced policies worse than random play. A governance process that accepted those self-scores would have approved fifteen regressions as improvements, and no internal signal would have flagged any of them.

The same paper proposes a remedy that matters for procurement language: a Sealed Exogenous Acceptance Loop (SEAL), which retains self-authored tests for the improvement signal but compares each candidate against the incumbent through a fixed harness-side audit the harness controls, not the agent. The mechanism is worth understanding because it generalizes. The failure was not that the agent wrote tests; it was that the agent’s tests were also the acceptance criterion. Separating the improvement signal from the acceptance decision is the structural fix, and it is a fix a reviewer can demand contractually.

The admissibility checklist

The evidence base converges on four requirements before a self-improvement score should count as capability evidence.

1. Held-out, contamination-sealed evaluation tasks. The improvement pool and the evaluation pool must be disjoint, and the held-out partition must be sealed against leakage. The agentic self-improvement survey prescribes a held-out distribution that never overlaps the optimization data. Community implementations show this is buildable: Tenacious-Bench, a synthetic benchmark for B2B sales agent alignment, was constructed with contamination-sealed held-out partitions and reports its own held-out delta (a trained judge scoring 58.7 versus a 42.3 baseline, +16.4 with p=0.003, all self-reported by the repository). That “contamination-sealed” is currently a self-attested property in such repos is itself part of the checklist: ask who verified the seal, because “we held out a test set” and “an independent party confirmed the agent never saw it” are different claims.

2. Benchmark lineage and version logging. A score is only interpretable if you know exactly what harness produced it. A pilot audit of twelve LLM agent benchmark papers, What Twelve LLM Agent Benchmark Papers Disclose About Themselves, found disclosure is systematically poor. On harness provenance, seven of eight agent benchmarks scored 0.5 on the audit’s schema and one, SWE-bench, scored 0.0; the recurring gap was that the environment image is not pinned by digest, so the environment a score was measured in cannot be reproduced exactly. On cost, none of the eight agent benchmarks report inference cost in any form, giving an author-reported cost-disclosure mean of exactly 0.00. If the field’s own benchmarks do not pin their environments, a vendor’s claim almost certainly does not either. Demand pinned harness and environment images, disclosed benchmark versions, and inference cost, or treat the score as non-reproducible.

3. Calibrated judges. Many improvement loops use an LLM as judge, which moves the trust question rather than answering it. The Evaluation Context Protocol proposes explicit floors: an LLM judge “should never be trusted until it has been calibrated against a minimum of 100 human-labeled examples” (ECP) and must reach Cohen’s kappa of at least 0.6 against human domain experts. These are protocol proposals, not audited standards, but they convert “our judge is good” into two checkable numbers. A claim that cannot produce calibration evidence for its judge has not finished its homework.

4. A representative improvement signal. Even honest evaluation cannot fix a loop trained on a skewed slice of reality. The Adaptive Data Flywheel deployment paper reports, by its own account, receiving feedback from 495 employees out of 30,000 users (flywheel paper), and its authors state this creates sampling bias that reduces the generalizability of the results. Ask whose feedback drove the improvement and whether it resembles the deployment population.

The comparison below compresses the checklist into the form a reviewer can take into a gate meeting.

The counterpoint worth taking seriously

The checklist above implies that reusing benchmarks is disqualifying. A September 2026 preprint, Which Self-Improvements Should We Trust?, argues the opposite can sometimes hold. Its proposed framework, Reuse, aims at a loop-wide statistical guarantee: for a user-specified level α, with probability at least 1−α, every promoted modification across the entire self-improvement process is a genuine population improvement on the underlying task distribution. In other words, reuse with controlled error rates, rather than mandatory fresh held-out sets. In the authors’ live self-improvement experiments, Reuse reduced the proportion of promotions that were false from up to 20.7 percent to 0 percent while finishing within 0.04 percentage points of the best baselines on final true population performance.

This is an author-reported, non-peer-reviewed preprint, and its method and quantitative results are unverified. But the argument changes the review conversation even if the specific framework does not survive scrutiny. A sophisticated vendor may respond to a held-out-set demand with “we control the promotion error rate instead.” That is a legitimate alternative only if its inputs are disclosed: what α was chosen, what task distribution the guarantee covers, and how the judge inside the loop was calibrated. “We control the error rate” is not the same claim as “we used held-out tasks,” and a reviewer should know which one is being offered and at what strength.

There is also evidence that improvement loops can produce real change, which sharpens rather than refutes the verification requirement. SAHOO, a recursive self-improvement alignment framework, was evaluated on 189 public-benchmark tasks (63 each from HumanEval, TruthfulQA, and GSM8K) and reports code generation and mathematical reasoning gains of 16 to 18 percent against truthfulness gains of only 3.8 percent, with truthfulness carrying a higher drift cost (a CAR of 0.60 versus around 0.67). A recursive self-improvement research agent, AIDE², tells the same story from the deployment side: in an autonomous 8-day run it discovered seven successive improvements that generalized to four held-out benchmarks, one of them out of distribution, and it kept the changes that performed best on hidden evaluations. All author-reported. The lesson for reviewers is twofold: loops are not inherently fake, and a single aggregate improvement figure can hide domain collapse. If a vendor reports one delta, ask for the per-domain breakdown.

Can a predictor replace the benchmark run?

For a reviewer the question is ultimately economic. Full held-out evaluation of every proposed self-modification is expensive, and teams will ask whether a learned model of improvement can stand in for most of those runs. Predict Before You Modify: Learning Improvement Dynamics for Self-Evolving Agents, an October 2026 preprint, is the most direct available test of that idea. Its Improvement Dynamics Model (IDM) is an editable program that forecasts task-level outcomes of a proposed code change before evaluation, uses the forecast to request at most one revision of the draft, and periodically rewrites its own forecasting code from past predictions and observed outcomes. Author-reported results span four domains (multilingual coding, paper review, robotics reward design, and Olympiad-level mathematical grading): up to 46 percent higher average performance over generations than the HyperAgents baseline, with gains that persist under matched token budgets and extend to 166 held-out coding tasks, where agents from predictive runs gained a median 13.25 percentage points over the shared initial agent against 5.42 for the baseline. Forecast quality improved as the predictor evolved, with final deployed versions reducing task-level prediction loss by an author-reported 8.7%–24.4%, though candidate predictor updates won their in-run comparisons in only 19 of 28 cases. Every number in that sentence is author-reported from a single non-peer-reviewed preprint whose code is released through an anonymized link.

The governance-relevant detail is what the predictor is not allowed to do. In this design the forecast never decides acceptance: “This evaluation proceeds regardless of the final predicted score.” The prediction only determines whether a draft gets one more revision pass (a median of 27 extra passes per coding run), and the claim that gains generalize rests on held-out tasks disjoint from modification and selection. The economics are also narrower than “cheap prediction replaces evaluation.” Complete 40-generation coding runs with the predictive method averaged 170.22 million tokens against 49.45 million for the baseline, author-reported, and the efficiency result is that at a matched budget equal to the baseline’s median full-run cost, predictive runs reached a 10.9 percent higher median area under the budget curve, a measure of how early strong agents are found. What prediction buys is better allocation of a fixed evaluation budget, not the elimination of evaluation.

The risk is the obvious extrapolation: if forecasts are this useful, why not let them gate promotion and skip the held-out run? Look at where the predictor’s knowledge comes from. It is trained on modifications and observed outcomes from the same run and domain, and in the coding setup the revision decision compares predicted and parent scores computed on the same ten preliminary tasks. That is sound while prediction only guides search, and the authors do validate each candidate predictor on later records before adopting it. Promoted to an acceptance gate, the same predictor becomes just another in-loop judge: its training evidence overlaps the evaluation it would gate, and a wrong forecast would be approved by a process that never touched an unseen task. The demand is the one the checklist already makes of every judge. Show that the predictor’s training records are disjoint from the benchmark it gates, and show its calibration, including how forecast accuracy drifts as the predictor itself is edited.

The two-gate rule: predict, then verify

The practical verdict follows directly. Reject in-loop scores, and reject predictor verdicts, as capability evidence. A learned predictor may triage: decide which drafts get another revision, which changes are worth evaluating first, where the evaluation budget goes. Errors at that gate cost tokens. Promotion into a deployed agent or capability-change documentation passes through the second gate, and there the requirements are unchanged:

1. Held-out, contamination-sealed evaluation tasks disjoint from the optimization data, with the seal’s verification stated (independent audit versus self-attestation).
2. Benchmark lineage logs: pinned harness and environment images, disclosed benchmark versions, and inference cost.
3. Judges calibrated against at least 100 human-labeled examples at Cohen’s kappa ≥ 0.6 versus human domain experts (ECP).

Two substitutions remain legitimate when their inputs are disclosed. A statistical-guarantee claim can replace fresh held-out tasks when α, the covered task distribution, and the judge’s calibration are disclosed (Which Self-Improvements Should We Trust?). And a predictor can carry part of the load when its training records are provably disjoint from the benchmark it gates and its calibration is documented; the ECP floors for judges are the right starting ask. Neither substitution appears in independently validated form in any study cited here. The strongest predictive method in this evidence set still evaluates every candidate it accepts into its archive.

Note what this rule does to incentives. A review process that accepts in-loop scores, or uncalibrated predictor verdicts, lets any team manufacture an improvement claim cheaply, which shifts the entire burden of proof onto auditors and raises the cost of legitimate capability evidence for everyone else. The gate is not just protecting one deployment; it is setting the price of the word “improved” across the organization.

What this evidence cannot settle

The limits here are substantial and should shape how confidently the checklist is applied. The anchor preprint and the predictor preprint each report their results in a single non-peer-reviewed preprint; the predictor’s code link is anonymized, its results cover four research domains and code-diff modifications to agent programs, and it tests no governance deployment and no predictor-based acceptance gate. Nearly everything supporting the checklist is likewise 2025 and 2026 arXiv preprints, plus one community repository reporting its own results. No study cited here validates the full checklist end-to-end in a real governance setting. The kappa ≥ 0.6 and 100-label thresholds are one protocol’s proposal (ECP), not an audited standard, and no result cited here has been independently replicated. The evidence also says nothing about regulatory obligations in any particular jurisdiction.

What the evidence does settle is narrower and still useful: self-authored and in-loop verification has been observed to report success while policies fail below random baselines, benchmark lineage is rarely documented well enough to reproduce a score, the methodological literature treats held-out evaluation as the test that separates capability from memorization, and the strongest predictive method in the cited evidence still evaluates every candidate before accepting it. Until independent replication arrives, that is enough to justify the gate. The claim that needs defending is not “require better evidence.” It is the vendor’s claim that the current evidence suffices.