DeepSWE, published July 8, 2026, is a 113-task benchmark built to answer an uncomfortable question: when a coding agent resolves an issue on a public leaderboard, is it engineering the fix or reproducing one it already saw in training? The benchmark’s design, per Datacurve, assumes you cannot tell the difference from SWE-bench-style scores alone, because those scores are earned against fixes that already sit in public GitHub history.
What does DeepSWE measure?
DeepSWE measures whether a coding agent can resolve an original, multi-file engineering task it has almost certainly never seen, graded against a hand-written test rather than an inherited GitHub patch. The benchmark ships 113 tasks written from scratch across 91 actively maintained open-source repositories and five languages: TypeScript, Go, Python, JavaScript, and Rust. None of the tasks are contributed back upstream, which keeps the reference solution out of the public record that training pipelines scrape.
The framing matters. SWE-bench and its descendants lift completed work from merged pull requests in public repositories. The issue text, the diff, and the test suite are all public artifacts that pre-date the evaluation. DeepSWE instead authors fresh tasks in repositories the model may know, but with fixes that have never existed in any public commit. If an agent solves them, it is doing the work, not recalling it.
Why contamination is a structural flaw in public coding benchmarks
A benchmark mined from public commits measures recall once the training set includes those commits. This is not a SWE-bench-specific problem. Any public coding benchmark that pulls from GitHub, commit logs, or shared code corpora carries the same exposure, and the exposure grows as trainers scrape more aggressively and as benchmarks sample more widely from the same sources.
The mechanism is straightforward. Model trainers scrape GitHub; benchmarks sample from GitHub; the two distributions overlap. A high score on a contaminated leaderboard is therefore compatible with two very different capabilities. One is genuine engineering: reading an unfamiliar codebase, forming a hypothesis, editing across files, and producing a correct change. The other is pattern completion: recognizing a known issue from its description and emitting a close approximation of the known fix. The leaderboard does not distinguish them.
DeepSWE’s contribution is not to prove that any particular model is contaminated. It is to build a leaderboard where the question is answerable in principle, because the ground truth is held out of public reach. That reframes what a coding-agent score is evidence of, and shifts the burden of proof back onto anyone citing a public number as a capability claim.
How does DeepSWE build and grade its tasks?
DeepSWE’s methodology combines original task authoring, hand-written behavioral verifiers, and a model-agnostic harness to isolate model capability from both contamination and scaffolding effects. Each task runs the Harbor task format: a task.toml metadata file, an instruction.md prompt, a pre_artifacts.sh setup script, an isolated Docker environment, and a solution/ directory holding the reference fix that is never consulted at grading time. The grading step uses verifiers hand-written to test the requested functionality and accept any implementation that provides it, rather than inherited test suites that can fail correct alternatives or pass incomplete ones.
Two design choices reinforce the long-horizon claim. DeepSWE’s prompts are roughly half the length of SWE-Bench Pro’s, yet their reference solutions touch 5.5x more code per task. Shorter spec, larger change. That inverts the usual relationship between prompt verbosity and task difficulty, and it is a useful sanity check against the criticism that “harder” benchmarks are just longer prompts wrapped around trivial edits.
The harness is a deliberate variable to control. DeepSWE grades through Pier, a Harbor-compatible fork, running mini-swe-agent as a model-agnostic interface. The intent is that leaderboard scores reflect model capability rather than scaffolding choices like per-vendor editing primitives or custom tool wrappers, which can dominate raw model differences on benchmarks that let each vendor bring its own harness. Repository selection enforces spread across the benchmark: public, actively maintained, at least 500 GitHub stars, permissive license, with the median repository contributing a single task so no codebase dominates the results.
A second data point sharpens the verifier argument. An independent LLM judge disagrees with DeepSWE’s verifier in 1.4% of trials, versus 32.4% for SWE-Bench Pro’s inherited tests. The 32% figure is worth pausing on. It means roughly one in three pass/fail decisions under SWE-Bench Pro’s grading is disputed by an independent judge. That is a separate failure mode from contamination: a benchmark can be clean and still be measuring the wrong thing if its oracle is unreliable, and an unreliable oracle can flatter weak models by passing incomplete fixes or penalize good ones by rejecting correct alternatives.
How do frontier models actually score?
The DeepSWE leaderboard, as of June 20, 2026, spreads models across a wider band than most public coding-agent leaderboards, where scores tend to cluster near the top:
| Model | DeepSWE Pass@1 |
|---|---|
| Claude Fable 5 | 70% ± 4% |
| GPT-5.5 | 67% ± 6% |
| Claude Opus 4.8 | 59% ± 2% |
| GPT-5.4 | 52% ± 2% |
Read the gap, not the absolute numbers. Claude Fable 5 and GPT-5.5 sit within overlapping error bars at the top; the three-point lead is not a decisive margin. Claude Opus 4.8 and GPT-5.4 trail by roughly 10 to 18 points. On leaderboards where contaminated single-issue fixes inflate the floor, that kind of spread compresses and the ranking signal weakens. The wider band here is consistent with, though not proof of, the contamination hypothesis: strip out the memorized fixes and the models separate more.
The error bars carry their own signal. GPT-5.5’s ±6% band against Fable 5’s ±4% means the first and second positions are statistically close, and the headline ranking is soft. Anyone procuring on the basis of a three-point lead is reading noise rather than a capability difference. The tighter bands on Opus 4.8 and GPT-5.4 reflect more samples and give those lower positions more positional certainty than the top of the table.
What does this mean for evaluating coding agents?
If public coding-agent leaderboards are partly measuring memorization, a vendor’s SWE-bench score becomes a weak signal of engineering capability, and the burden of verification shifts to the buyer. Benchmark-driven procurement assumes the benchmark measures capability. Contamination breaks that assumption, and once it is broken the whole logic inverts: a jump on a public leaderboard could be new capability, or it could be new coverage of commits that were previously held out.
That is expensive to fix. Authoring original tasks, writing behavioral verifiers, and standing up isolated Docker environments per task is exactly the labor DeepSWE’s authors undertook to produce 113 items. Most engineering organizations will not replicate that rigor internally. The realistic outcome is stratification. Large platform teams build and maintain private eval suites against code the vendor cannot have seen; smaller teams rely on public numbers and absorb the contamination risk, or fall back on trial periods and gut feel. The cost of a defensible evaluation rises, and it rises unevenly.
The benchmark also reframes how to read vendor releases. When a model posts a jump on a contaminated leaderboard, the honest question is how much of the gain is new capability versus new coverage of previously held-out commits. DeepSWE does not let you answer that for a specific model on its own terms, because the comparison is across benchmarks, not within one. But it makes the question legible, and it gives buyers a held-out reference point to triangulate against.
Will DeepSWE itself stay uncontaminated?
DeepSWE’s clean-measurement property is durable only as long as its reference solutions stay out of public reach, and no public benchmark has held that line permanently. The held-out guarantee depends on solutions never leaking into GitHub. That is enforceable today, while the benchmark is young and the repository list is curated. It is harder to enforce once the benchmark is well-known and widely cited.
The same saturation pressure that eroded SWE-bench applies here. Once task descriptions, repository names, and prompt templates circulate, they enter the training corpus and the held-out property degrades. A model trained after DeepSWE’s release may have seen the task framing even if it has not seen the solution, which is a softer form of contamination than a memorized diff but a real one. A sufficiently capable model can also reconstruct a held-out solution from a recognizable task spec, at which point “held-out” stops meaning “unseen.”
The honest read is that DeepSWE buys a window of clean measurement, not a permanent one. The methodology, hand-written verifiers and behavioral testing against original tasks, is the durable contribution. The specific 113 tasks will age, and the leaderboard they produce will saturate on the same schedule as every public benchmark before it. The next useful benchmark will likely look like DeepSWE in method and share none of its tasks, which is the point: held-out evaluation is a moving target, and any fixed leaderboard eventually becomes a memory test.
Frequently Asked Questions
Can DeepSWE rank models within a single programming language?
No. DeepSWE spans TypeScript, Go, Python, JavaScript, and Rust, but the 113 tasks are spread across 91 repositories and the median repo contributes only one task. The per-language samples are too thin for reliable language-specific rankings, so it works best as a cross-language signal rather than a per-language leaderboard.
What is the minimum cost to run a DeepSWE-style private evaluation?
The cost floor is the labor to author each original task, hand-write a behavioral verifier, and maintain an isolated Docker environment, repeated across enough tasks to tighten error bars. For DeepSWE’s 113 tasks across 91 repositories, most teams cannot match that density, so the realistic choice is between a smaller held-out suite or continuing to rely on public benchmarks and accepting the contamination risk.
Could DeepSWE’s verifier disagree with what a human reviewer would accept?
Yes. A 1.4% disagreement rate with an independent LLM judge only shows the verifier is internally consistent; it does not prove the verifier catches every edge case a human or production test would care about. SWE-Bench Pro’s 32.4% disagreement rate is the warning: even a consistent oracle can be noisy, so teams should still map tested behavior to their own operational requirements.
Should a team retire SWE-bench once DeepSWE is available?
No. SWE-bench remains a useful coarse, widely-cited reference, but it should not be the only procurement signal. The safer workflow pairs public benchmarks with private held-out tasks drawn from your own codebase, where no vendor could have seen the fixes during training.
