Beyond SWE-bench: How to Test Coding Agents on Your Own Codebase

If you are deciding whether to widen a coding agent’s access to your monorepo, the cheapest useful evidence is no longer a public SWE-bench leaderboard score. It is a benchmark built from your own repository’s test execution, filtered by the validity checks the latest task-synthesis pipelines build in: a task must fail when the relevant code is masked, and its tests must fail-to-pass. Near-duplicate screening is the third filter you need, and it is the one you design yourself; no pipeline in this family ships it. Four papers published across 2025 and 2026 describe working versions of this machinery, most recently SWE-Nexus (observed 2026-10-16), which composes repository-level tasks directly from test runs. The practical consequence is a shift in where the cost sits. Authoring evaluation tasks used to be the bottleneck; validating them now is. One caveat up front, because it shapes everything below: every number in this article is author-reported in a preprint, and no independent replication appears in any of the papers.

Why the leaderboard score doesn’t transfer

Distrust of public coding-agent leaderboards usually gets framed as a vibe: contamination, gaming, unfamiliar repos. The stronger point is that leakage risk is documented inside the SWE-bench family itself. SWE-bench-Live makes recency its contamination control: “To ensure freshness and reduce the risk of data contamination from pretraining, we restrict the dataset to issues created between January 1, 2024, and April 20, 2025.” The initial release accordingly consists of “1,319 task instances created between January 2024 and April 2025, spanning 93 real-world repositories” (SWE-bench-Live). The paper states the reason plainly: “To reduce the risk of data leakage, SWE-bench-Live prioritizes recency by including only issues created after January 2024 in our initial release.”

That design choice is an admission. If the family that owns the most-cited leaderboard treats pre-2024 issues as contamination-suspect, an engineering lead is on firm ground declining to read an older public score as evidence about their own codebase. The same caution shows up in the task-generation literature: R2E-Gym removed repositories overlapping SWE-Bench test sets before running experiments, cutting its pool from over 8.1K problem statements to 4,578 (R2E-Gym’s Table 1). Decontamination is not a paranoia; it is standard practice among the people building these benchmarks.

So the question for a lead is not “which public score do I trust” but “what would a benchmark built from my own repository actually measure, and what would it cost to build one.” The second half of that question is what changed in 2025 and 2026.

Four pipelines that turn a repository into tasks

The newest entry, SWE-Nexus, describes itself as “a fully automated pipeline that constructs repository-level implementation tasks from test execution. Four construction agents select tests, adapt tracing and evaluation protocols to each repository, and generate behavioral specifications.” Its published benchmark contains 600 tasks, 100 per language (the paper), with reported statistics on task composition, difficulty by language, repository coverage, and scoring points per task.

The other three approaches differ mainly in what raw material they start from:

• SWE-smith synthesizes bugs inside existing repositories: “generating errant rewrites of functions with an LM, (2) procedurally modifying the abstract syntax tree (AST) of functions, (3) undoing PRs, and (4) combining bugs.”
• R2E-Gym generates environments “using only commits from GitHub repositories, reducing reliance on both human-written issues and test cases,” via a recipe called SweGen that uses backtranslation and test generation.
• SWE-rebench pushes the scale-first end of the design space: “After applying all filtering criteria, approximately 153,400 potential task instances remain” (SWE-rebench). Read that number carefully. It counts filtered potential instances, not deduplicated or validated tasks.

The validity checklist is the actual product

What these papers converge on, and what transfers to an internal rollout regardless of which pipeline you imitate, is a short list of filters that separate a real task from a fake one.

1. A task must fail when its subject is removed. SWE-Nexus states the rule directly: “A scope is retained only if all expected scoring points pass in the original repository and masking the traced bodies causes at least one failure or error.” This is the strongest built-in check in the family, because it targets two failure modes at once: leaky tests (a test that passes whether or not the agent did anything) and tasks solvable without touching the relevant code. If masking the traced function bodies breaks nothing, the task cannot measure implementation competence, and it gets discarded.

2. Tests must be fail-to-pass. SWE-smith’s filter: “We apply each candidate patch to the corresponding repository, run the test suite, and only keep patches that break one or more existing, passing tests (referred to as Fail-to-Pass or F2P test(s)).” A two-minute testing runtime cap discards bug candidates that exceed it, which keeps the loop cheap enough to run repeatedly. For an internal suite, this is the cheapest template to copy: mutate code, run your existing tests, keep only mutations that flip a green test to red.

3. Screen for contamination and near-duplicates. R2E-Gym’s practice on public code was repo-level: “We decontaminate this set by removing repositories overlapping with SWE-Bench test-set repositories, obtaining 4578 problems (referred to as R2E-Gym-Subset)” (R2E-Gym). Inside a private monorepo the contamination concern is different (your agent vendor’s training data is unlikely to contain your code), but the near-duplicate concern is not. Hundreds of tasks generated from the same hot file will overstate whatever they measure. The decontamination these papers document targets benchmark overlap specifically, not general near-duplicate detection, so the deduplication step is one you design yourself.

A working checklist for an internal benchmark, assembled from the pipelines’ own stated filters:

• Does the task fail when the relevant code is masked or stubbed? If not, drop it.
• Is every scoring test fail-to-pass against the unmodified repo? If not, drop it.
• Is the task a near-duplicate of another task in the pool? If so, collapse them.
• Can the task be solved without reasoning about the code it claims to test? The masking check answers this mechanically; spot-check a sample by hand.

Where the automation ends

“Fully automated” deserves scrutiny, because the same research lineage contains the counterexample. In SWE-Gym, executable environments required manually configuring dependencies for each task instance, using configuration files, CI scripts, or repository documentation. The paper describes the newer alternative as: “Instead of manual configuration, they employ a general environment setup script and simply discard instances that fail the setup process.”

Note what that tradeoff does. Discarding failed setups is cheap, but it biases the surviving task pool toward repositories with cooperative build systems. If your monorepo has idiosyncratic dependency wiring, a general setup script may silently select against exactly the parts of your codebase you most want evaluated. Before trusting yield numbers, probe where environment setup fails on your repository and whether the failures are random or systematic.

Language coverage is the other boundary. SWE-smith is explicit about its center of gravity: “The mechanisms to identify programmatic objects (e.g. functions, classes) and perform transformations rely heavily on the Python specific ast library.” The authors describe the strategy as transferable, but the tooling as published is Python-first. SWE-Nexus reports 100 tasks per language across its 600-task benchmark (the paper), which is a broader claim, but the paper reports no per-language validity rates, so there is no evidence about whether masking-based retention works equally well across the six languages it covers. If your monorepo is primarily TypeScript, Go, or Java, treat multi-language support as a claim to verify on a small pilot, not a given.

What a synthesized benchmark can honestly measure

All four pipelines produce single-shot patch tasks: given a repository state and a specification or failing test, produce a change. Real agent usage is not that. The SWE-Touch study of actual sessions found that “59.0% of sessions contain repository changes attributed to the user (Figure 1(b)). This prevalence establishes code edits as an important interaction channel alongside messages: each edit changes the program state observed by subsequent agent actions.”

A benchmark synthesized from static test runs never exercises that channel. It measures whether an agent can produce a correct patch in isolation, not whether it behaves well when a human is editing the same files between turns. For a rollout decision, the honest reading is narrower than the marketing version: a synthesized internal benchmark supports before/after comparison inside your own repository. It answers “did agent version B get better than version A on tasks derived from our tests?” It does not answer “is this agent competent on our codebase” in any transferable sense, and it says nothing about interactive behavior.

A rollout protocol

Putting the pieces together, the protocol I would run before widening agent access looks like this:

1. Generate candidate tasks from your own test execution, following the SWE-Nexus pattern if your tooling supports tracing, or the SWE-smith pattern (inject bugs, run the suite) if you want the cheapest start and your repo is Python.
2. Apply the validity filters before looking at any score: masking must induce a failure, scoring tests must be fail-to-pass, near-duplicates get collapsed. Budget your engineering time here, because this is where the cost now lives.
3. Watch the environment-setup discard rate. If automated setup fails disproportionately on some services or modules, your benchmark is measuring the cooperative parts of your repo. Say so in the rollout document.
4. Run agents before/after, not against a public bar. The deliverable is a relative comparison inside your monorepo: which agent, model version, or prompt configuration performs better on your tasks. Do not publish or circulate the number as a competence claim; the evidence base does not support that reading.
5. Complement with a sample of interactive sessions if the user-edit channel matters to your rollout, since synthesized single-shot tasks exclude it by construction.

Limitations

Everything quantitative here is author-reported in preprints, and the strongest limitation applies to the peg itself. The SWE-Nexus paper describes pipeline mechanics and aggregate benchmark statistics (600 tasks, 100 per language) but reports no per-language validity rates for the masking-based retention check. No independent replication of any of these numbers appears in the preprints. The ~153,400 SWE-rebench figure counts potential instances after filtering, not validated tasks (SWE-rebench). Decontamination in these pipelines targets SWE-Bench repository overlap, not general near-duplicate detection. And none of these pipelines measures the interactive user-edit behavior that SWE-Touch documents in 59.0% of real sessions.

None of that makes internal task synthesis a bad bet. It makes it a bet whose payoff is a cheaper, repo-specific relative measurement, with validity checks you can run yourself. That is considerably more useful than a leaderboard score you have documented reasons not to trust, and considerably less than a guarantee.

Frequently Asked Questions

What is a limitation of synthesized benchmarks regarding interactive behavior?

A benchmark synthesized from static test runs never exercises that channel. It measures whether an agent can produce a correct patch in isolation, not whether it behaves well when a human is editing the same files between turns.