Can Coding Agents Reproduce Deep Learning Bugs? A Triage Guide

A coding agent can take the first pass at reproducing a deep learning bug report, and the newest evidence says the first pass can work often enough to be worth routing. The catch is that the strongest result so far does not come from better generation. It comes from verification. The RepGen preprint reports that its context-aware generation pipeline, wrapped in an iterative verification loop, reproduced 85 of 106 real-world deep learning bugs, a study-reported 80.19%. The same paper notes that manual reproduction of DL bugs succeeds only about 3% of the time (a figure it cites from a prior study, so treat it as secondhand). If you maintain an ML library and triage dependency bug reports, the practical shift is this: generation is cheap, so your bottleneck moves to whether you have a deterministic fail-to-pass signal that can grade whatever the agent produces.

That reframe changes triage from generate-and-hope into generate-then-verify. The rest of this guide walks through what the published numbers actually measure, where agents demonstrably stall, and a decision rule for when handing the first reproduction pass to an agent saves you days versus when it burns them.

Why reproduction, not diagnosis, eats the triage budget

Deep learning bugs are unusually hostile to reproduction. They depend on framework versions, hardware, random seeds, and training-state details that reporters rarely include. RepGen’s authors put the manual success rate at roughly 3% of examined DL bugs, citing an earlier study, which is the pain this whole line of research targets. A maintainer staring at a vague “loss goes NaN after upgrade” report is looking at hours or days of environment archaeology before any diagnosis can start.

That cost is why an automated first pass is attractive even at partial success. If an agent converts even half of incoming reports into runnable, failing scripts, the maintainer’s job shifts from reproduction to diagnosis, which is the part that actually requires their knowledge of the library.

What RepGen actually reports, and what 80.19% measures

RepGen’s headline number needs its machinery attached. Per the paper, the tool generates candidate reproduction code, then “iteratively refines the candidate based on four types of verification feedback, including syntax, code quality, semantic relevance, and approximate failure behaviour.” The 80.19% figure (85 of 106 curated real-world DL bugs) is the output of that loop, not of a single-shot prompt. Strip the verification away and you do not have RepGen; you have a code generator with good context.

The paper also reports a controlled study with 27 developers: using the tool improved reproduction success by 23.35% and cut reproduction time by 56.8%. Those are author-reported results from a controlled setting, not field data, but they point at the right mechanism. A tool that hands a human a script plus a verification verdict compresses the part of triage that is mechanical and leaves the part that is judgment.

Two cautions belong next to the number. First, 106 bugs is a curated dataset, and none of the other studies discussed here replicates the result. Second, RepGen’s verification is hybrid, not a deterministic oracle. Syntax parsing, bytecode compilation and Pylint checks are static, but the semantic-relevance and approximate-failure-behaviour signals come from LLM feedback, and the final reproduced-or-not verdict is issued by an LLM. The authors state the consequence in their limitations: “Verification and reproduction assessment can lead to false positives sometimes. Static checks cannot establish that the reported failure occurred, and LLM assessment can misinterpret runtime output, particularly for silent bugs.” The 80.19% therefore rests partly on model judgment, and the comparison numbers below show what changes when the grading is deterministic instead.

The rate landscape, every figure study-reported

Reproduction rates across recent studies vary by more than 40 points, and the variation tracks domain and oracle structure more than model quality. All figures below are paper-reported on curated benchmarks with named models; each of these papers reports its own benchmark, and none replicates another’s results.

Read down the oracle column and a pattern emerges, with one split worth making. iCoRe, a retrieval-augmented pipeline, reports Fail-to-Pass rates of 42.0% on SWT-bench Lite and 52.8% on TDD-bench Verified with GPT-4o, with relative improvements of 19.7%–31.7% over the best prior retrieval methods, and its grading is execution-deterministic: a reproduction counts only if a generated test “fails on the buggy version, passes on the fixed version, and introduces no new failures on the fixed version.” RepGen’s higher 80.19% is not graded that way; it rides on the hybrid check described above, whose final verdict is an LLM’s.

The last row is the sharpest warning. AgentBug-Smith reports that state-of-the-art agents resolve at most 9.00% of Live-Harness-Bench bugs, against 40.67% on SWE-Bench Verified, a drop of more than four times. If your repo’s bug reports are about training-loop plumbing, distributed setup, or CI harness behavior, the 9.00% side of that comparison is the more honest prior.

Accurate plans do not rescue execution

The most useful counter-evidence to “agents are getting good at this” comes from BugCraft, an end-to-end crash-reproduction system for Minecraft bug reports. Its evaluation found that “Even when provided with accurate step-by-step plans from the Step Synthesizer, the Action Model still failed to reproduce the bug in a majority of cases (56.14%).” The dominant failure causes were capability failures, not planning failures: “Poor Decision Making” at 29.82% and “Stuck in Loops” at 19.30%.

This matters for a specific reason. If you assume a stronger prompt or a better planner closes the gap, BugCraft’s data says otherwise: hand the agent a correct plan and it still fails most of the time on its own. The bottleneck is execution discipline, the ability to notice a dead end and stop retrying it. A verification loop supplies exactly that discipline from outside, which is a structural argument for why RepGen’s design outperforms end-to-end agent attempts, and a warning against reading 80.19% as a general agent capability. It is a property of a curated DL dataset plus a tight verify-and-refine loop.

Contamination and cost: the two quieter findings

Two findings from the edges of this literature deserve a place in your triage thinking.

Data contamination was checked by two of these studies, and both found the effect limited. Libro evaluated its Java reproduction pipeline on 31 bug reports submitted after the LLM’s training data collection ended, and still produced bug-reproducing tests for 32% of them, essentially matching its 33% overall rate; it is the only system here evaluated directly on reports filed after its model’s training cutoff. iCoRe checked the threat with two analyses on TDD-bench Verified: performance across 2017–2023 issue-creation years stayed relatively stable (Mean 57.16%, SD 11.75%), and textual similarity between its generated tests and existing tests stayed below 0.56 for 90% of instances, leading its authors to conclude “the impact of data contamination is limited.” Neither check settles whether RepGen’s 80.19% survives contact with bugs filed after its models’ cutoffs: 31 reports is a small sample, and year-stability and similarity scores are indirect signals rather than post-cutoff filings. As a review habit, though, asking “was any of this evaluated post-cutoff?” is now cheap and worth asking of every reproduction paper.

Cost is no longer the argument against a first pass. LTGDroid, an Android reproduction system, reports an average of 20.45 minutes and approximately $0.27 to successfully reproduce a single bug. Those are author-reported figures for successful reproductions, so the true per-bug cost including failures is higher, but the order of magnitude is clear: the agent’s first pass costs less than the first ten minutes of a maintainer’s time. Money and compute are not the constraint. The oracle is.

The cost that moved: writing the verification oracle

Here is the consequence the headline rates obscure. AgentBug-Smith reports that less than half (47.11%) of the issues it studied contain in-patch tests, a rough proxy for how often a ready-made verification signal exists at all. For the other half, someone has to write the fail-to-pass check before an agent’s output can be trusted, and none of these six studies measures what that authoring costs a maintainer.

So the economics of agent-assisted reproduction are asymmetric. Where a deterministic oracle already exists (a test harness, a reproducing test in the patch, a crash signature), the agent’s first pass is nearly free and the published rates in that regime run from a third to about half of curated bugs depending on domain. RepGen’s 80.19% sits above that band on an LLM-assisted check rather than a deterministic one, so the higher number buys you a strong lead, not a guarantee. Where no oracle exists, the agent generates plausible-looking scripts that you must verify by hand, which re-creates the original triage cost with extra steps. Generation got cheap; verification did not.

A maintainer’s triage decision rule

Pulling the studies together, I would route incoming reports like this:

1. Does a deterministic fail-to-pass signal exist or can you write one in under an hour? If yes, let the agent take the first pass and grade its output against the oracle. This is the regime iCoRe’s 42.0%–52.8% study-reported rates come from. RepGen’s 80.19% is the adjacent encouraging number, but it is graded by the hybrid LLM-assisted loop, so treat a RepGen-style verdict as a strong lead that still needs confirming against a real check.
2. Is the bug in patch logic or in harness and plumbing? Harness bugs are where agents collapse to at most 9.00% resolution per AgentBug-Smith. Route those to a human first, or expect the agent’s output to need full manual review.
3. No oracle and no cheap way to build one? Treat agent output as a hypothesis generator, not a reproduction. BugCraft’s 56.14% failure rate with correct plans in hand is the documented behavior of agents acting without tight feedback.
4. How vague is the report? Every benchmark here uses curated reports. None of the six studies measures performance on the one-line “it broke after I upgraded” issues that dominate a real tracker, so assume degradation and keep a human in the loop for acceptance.

This decision rule is an inference drawn across six studies, not a result any single paper states. The reasoning: every headline success leans on a signal that grades candidates against the reported failure, whether a deterministic test or an LLM’s approximation of one, so verification, not generation, is the binding constraint, and the 47.11% in-patch-test statistic quantifies how often that constraint goes unmet in real repositories.

What no benchmark here can tell you

The limits deserve plain statement. Every rate cited above is paper-reported on a curated dataset: 106 DL bugs, 750 Defects4J cases, 86 Minecraft reports, SWT-bench and TDD-bench slices, and Live-Harness-Bench. None of the six papers replicates another’s results. The ~3% manual reproduction baseline is a citation inside RepGen, not a verified figure. Libro’s contamination control covers 31 reports, and iCoRe’s relies on stability and similarity proxies rather than post-cutoff filings. None of these six studies measures the maintainer cost of authoring the verification oracle, which this guide argues is the real deciding expense, and none measures behavior on vague or incomplete real-world reports.

What survives those limits is the procedure, not the percentages. Generate candidates, grade them against verification signals, refine, and only then hand the result to a human. That structure held across every strong result in these six papers, and it will still hold when the current model names and scores date out of them. If your repo can answer “did this script reproduce the failure?” with a deterministic check, an agent’s first reproduction pass is worth routing. If the only check is an LLM’s judgment of the output, the pass is still worth running, but the verdict needs a human before it counts as a reproduction. If there is no check at all, the published rates are describing somebody else’s benchmark, not your tracker.

Frequently Asked Questions

What is the cost to reproduce a single bug using an agent?

LTGDroid, an Android reproduction system, reports an average of 20.45 minutes and approximately $0.27 to successfully reproduce a single bug. Those are author-reported figures for successful reproductions, so the true per-bug cost including failures is higher, but the order of magnitude is clear: the agent’s first pass costs less than the first ten minutes of a maintainer’s time.

How often do bug reports include a ready-made verification signal?

AgentBug-Smith reports that less than half (47.11%) of the issues it studied contain in-patch tests, a rough proxy for how often a ready-made verification signal exists at all. For the other half, someone has to write the fail-to-pass check before an agent’s output can be trusted, and none of these six studies measures what that authoring costs a maintainer.

What happens if an agent has a correct plan but no tight feedback?

BugCraft’s 56.14% failure rate with correct plans in hand is the documented behavior of agents acting without tight feedback.