groundy
Agents & Frameworks

Reviewing AI Agent Code in Open Source: A Maintenance Checklist

A 2026 study shows file context predicts agent code persistence better than code tokens, supporting a five-gate checklist to track post-merge maintenance costs.

Published 5 references
A skeptical green resin dinosaur wearing a black ruff adjusts a yellow scale on its curled tail, with an open green arch behind it on a warm ivory background.
On this page12 sections

A 2026 arXiv study followed the lines added by 2,802 merged pull requests (the survival study), 1,470 of them attributed to agents and 1,332 to humans, across 215 open-source repositories, from the moment each line merged until it was first modified or the observation window ended. The finding that matters for review practice: whether agent-attributed lines get touched again is better predicted by the file they land in than by the code they contain. A file’s size and recent changes reached AUC-ROC 0.78, against 0.70 for the code’s own tokens. Meanwhile, the acceptance number most teams report is defined to stop at the merge. A 2019 empirical study drew the boundary the same way, and a separate 2026 analysis still defines acceptance as the proportion of merged PRs (arXiv:1908.09321, arXiv:2602.08915v2). Everything that happens afterwards, the rewrites, the verification work, the ownership handoff, is invisible to that metric.

This article turns that gap into a review checklist. The short version: report submission, acceptance, merge, line-level persistence, and follow-up maintenance as five separate gates, because the evidence shows they measure different things. The longer version below covers what each stage’s evidence actually supports, what it cannot settle, and where the numbers come from, since every figure here is tied to one dataset.

Where “accepted” stops: the merge boundary

The definition problem is old and precise. In their study of code quality and pull request acceptance, the authors wrote that “We identified whether a pull request was accepted or not by checking whether the pull request had been marked as merged into the master branch or whether the pull request had been closed by an event that committed the changes to the master branch” (arXiv:1908.09321). This is a reasonable operationalization, and the same merge boundary still governs: the separate 2026 acceptance analysis defines its rate as the proportion of PRs with a non-null merged_at timestamp (arXiv:2602.08915v2). But notice what it encodes: acceptance is a property of the pull request’s lifecycle, not of the code’s life. The metric terminates at the boundary where the change enters the branch. It carries no information about whether those lines survive a week, a quarter, or the next refactor.

A separate 2026 analysis of AIDev shows why leaning on acceptance alone is doubly shaky: the rate itself moves with task type. Comparing five agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) across 7,156 pull requests from the AIDev dataset, the study found (arXiv:2602.08915v2) that “documentation tasks achieve 82.1% acceptance compared to 66.1% for new features—a 16 percentage point gap that exceeds typical inter-agent variance for most tasks.” Read that carefully. Which agent tops a leaderboard can depend more on the task mix it was given than on the agent. An acceptance dashboard that does not stratify by task type is mostly reporting your ticket queue’s composition.

The trend numbers deserve the same skepticism. In that analysis, Devin was the only agent of the five showing a consistent positive acceptance trend, +0.77% per week over 32 weeks, while the others remained largely stable (arXiv:2602.08915v2). When a vendor claims broad, steady improvement, this is a useful calibration point: over eight months of observed data, four of five agents did not move much at all.

The survival study: population and how code fate was defined

The post-merge work measures something different. As the survival study describes it: “We follow the lines added by 2,802 merged PRs, 1,470 agent-attributed and 1,332 human, in 215 open-source repositories from AIDev, from the merge until the line is first modified or the observation window ends.” That is a line-level survival definition, not a PR-level one. A pull request can be partially rewritten; the metric sees which lines persisted.

The population sits inside AIDev, a dataset of 932,791 pull requests opened by five agents on GitHub (arXiv:2601.16809). The observation window is explicit: AIDev contains PRs up to August 1, 2025, lines still unmodified on December 31, 2025 are right-censored, and every line is observed for at least five months. Attribution is audited rather than assumed. The authors manually inspected 325 agent PRs, had a second rater independently relabel 100 of them (Cohen’s κ = 0.85), and found the agent made substantive changes in 87.3% of the audited PRs in the cohort. The one cohort detail the paper does not enumerate is the language mix across the 215 repositories, which is worth a check in the full text before you generalize the numbers to your own stack.

What predicts a second touch: the file, not the tokens

Here is the result with the most direct review consequence. The study reports that “a file’s size and recent changes predict whether agent-attributed code will be modified better than its code tokens do (AUC-ROC 0.78 vs. 0.70)” (arXiv:2601.16809).

Two readings follow. The first is operational: if you want to know which agent-authored changes are likely to need attention later, look at where the diff lands. A large file with heavy recent churn is more likely to be edited again, and that signal outranks anything about the code’s textual content. Routing review attention by file context is cheap; you already have file size and commit history in your repository metadata. Keep the base rate in view, though: 79.7% of the agent-attributed files in the study’s forecasting dataset were modified anyway, so file context ranks follow-up risk rather than detecting it (arXiv:2601.16809). The second reading is cautionary: modification is not failure. Files churn because they are central, not necessarily because the incoming code was bad. The predictor tells you where follow-up work will concentrate, not whose code was worse.

Does agent code get modified sooner?

The headline survival comparison is conditional in both directions. Within the same repositories, agent-attributed code is not modified sooner than human code in any design the study examined. When each PR is weighted equally, agent-attributed lines are modified later (HR = 0.83, 95% CI [0.75, 0.93]), and the difference comes from the supervised assistants, mainly Copilot and Codex (HR = 0.76); Devin does not differ from human code in the full cohort (HR = 0.95). By the end of the window, 28.0% of agent-attributed lines and 34.5% of human lines were modified (PR-weighted), and without the weighting no difference is statistically significant (arXiv:2601.16809). The authors’ own summary is the right calibration: “Our findings suggest that agent code is not disposable, but whether it survives longer than human code depends on the agent, on when the code was merged and on how the comparison is computed.”

The most interesting counter-result cuts against the intuitive narrative. When agent-attributed code is modified, the intent of the change is similar to that for human code, with Cramér’s V = 0.080 (arXiv:2601.16809). Cramér’s V near zero means weak association; in plain terms, the mix of reasons people edit agent lines looks roughly like the mix of reasons they edit human lines. Later changes resemble ordinary maintenance more than targeted cleanup of agent output. The intent result stays conditional on modification: it tells you what later changes look like, not why one group’s lines last longer.

A five-gate maintenance checklist

The stages measure different things, so report them separately:

GateWhat it answersEvidence anchorWhat to log
1. SubmissionWhat did the agent open, and how is authorship attributed?AIDev labels PRs opened by five agents; 932,791 PRs total (study)Attribution method, task type, target file
2. AcceptanceDid the project take the change?Docs 82.1% vs. new features 66.1%; 16-point gap (study)Acceptance rate stratified by task type, not blended
3. MergeWhen did the code become the project’s problem?Acceptance defined as merged into master or closed by a committing event (study)Merge timestamp, review latency, PRs accepted but stalled
4. PersistenceHow long do the lines survive unchanged?Line tracking from merge to first modification (study)Time-to-first-modification per line, with file size and recent churn as covariates
5. Follow-up maintenanceWhat does keeping the code cost after it survives?Verification case study (study)Verification effort, protocol owner, follow-up PRs

Gates 4 and 5 are the ones your dashboard almost certainly lacks, and they need different instrumentation. Persistence comes from your version control history: for each line an agent adds, record when it was first modified and what the enclosing file looked like at merge time. Follow-up maintenance is messier, because much of it never becomes a PR.

The coding-theory case study is the clearest documented example of that cost. Over five weeks, “agent sessions found genuine improvements to published bounds in coding theory, and also recorded wrong intermediate conclusions that hid one of those improvements for three weeks” (arXiv:2609.39081v1). In that setup, the agent wrote the verifiers and the search code, while a human chose the problem and set the verification protocol. Three weeks of delay from a wrong intermediate conclusion is exactly the kind of loss that no acceptance or merge metric will ever register. It has to be logged deliberately, with a named owner for the verification protocol.

One caveat on the checklist itself: the evidence shows the gates measure different things. It does not demonstrate that instrumenting them improves outcomes. Treat the scorecard as better measurement, provisionally, not as a proven intervention.

How much agent tooling is wired into what you measure

Before building any of this, it helps to know where agent output actually enters your repositories. A dataset study (arXiv:2605.08435v1) catalogued “15,591 configuration artifacts from 4,738 repositories across five tools and eight configuration mechanisms, together with the full content of 18,167 configuration files.” That work measures adoption footprint, not code fate, so it does not tell you anything about persistence. Its value here is scoping: agent tooling is configured through many mechanisms, and an internal audit of which mechanisms exist in your repos is the precondition for trusting any authorship attribution you later compute.

What this evidence cannot settle

Population boundaries first. The acceptance analysis (arXiv:2602.08915v2) covers a different slice of AIDev than the survival study (arXiv:2601.16809): 7,156 pull requests, eight months, repositories with 100+ stars, versus 2,802 merged PRs across 215 repositories. Do not quote a number from one and a number from the other as if they described the same population. The acceptance authors state it directly: “Our data covers only 8 months from 100+ star repositories (external validity), limiting generalization to smaller projects” (arXiv:2602.08915v2).

The survival study’s authors (arXiv:2601.16809) are equally blunt: “Our findings may not generalize beyond our 215 repositories, for example to small or closed-source projects, or to agents released after August 2025.” That last clause matters most. Everything here reflects agents available through August 2025. Newer models may behave differently, and the honest response is to re-baseline rather than treat 0.78 or 82.1% as constants (arXiv:2601.16809, arXiv:2602.08915v2).

Finally, attribution itself is a soft spot, though an audited one. The study’s manual check found the agent made substantive changes in most audited cohort PRs, and dropping the PRs the audit labeled human leaves the headline estimate unchanged; even so, human PRs can contain undisclosed AI-assisted code, so the comparison is agent-attributed versus not-attributed (arXiv:2601.16809). If your team mixes agent-initiated PRs with human PRs that used agent assistance, your persistence comparison is measuring authorship classification as much as code quality.

What I would instrument first

Start with gates 2 and 4, because they are cheapest and most distorted today. Stratify acceptance by task type; the 16-point documentation-versus-features gap means a blended rate is close to meaningless (arXiv:2602.08915v2). Then compute time-to-first-modification for agent-added lines alongside file size and file churn, so persistence is always read in context (arXiv:2601.16809). Add gate 5 as a manual log at first, a line per project noting who owns verification and what it cost, because the three-week hidden improvement in the coding-theory case is the failure mode no automated metric caught (arXiv:2609.39081v1).

Every threshold you derive from these numbers is provisional until the survival work is replicated on another dataset, and the language mix across the 215 repositories is the one cohort detail worth checking in the full paper before you commit a quarter’s metrics program to it. The durable move is structural, not numerical: stop treating the merge as the finish line, and start reporting what the code cost after it crossed.

Frequently Asked Questions

What predicts whether agent-attributed code will be modified after merge?

a file’s size and recent changes predict whether agent-attributed code will be modified better than its code tokens do (AUC-ROC 0.78 vs. 0.70)

How does acceptance rate vary by task type in the AIDev dataset?

documentation tasks achieve 82.1% acceptance compared to 66.1% for new features—a 16 percentage point gap that exceeds typical inter-agent variance for most tasks.

Does agent code get modified sooner than human code?

Within the same repositories, agent-attributed code is not modified sooner than human code in any design the study examined. When each PR is weighted equally, agent-attributed lines are modified later (HR = 0.83, 95% CI [0.75, 0.93]), and the difference comes from the supervised assistants, mainly Copilot and Codex (HR = 0.76); Devin does not differ from human code in the full cohort (HR = 0.95).

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. the survival studyarxiv.orgAccessed
  2. arXiv:1908.09321arxiv.orgAccessed
  3. arXiv:2602.08915v2arxiv.orgAccessed
  4. arXiv:2609.39081v1arxiv.orgAccessed
  5. arXiv:2605.08435v1arxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy