A study of 278,790 code review conversations across 300 GitHub projects found that AI agents supplied 88,011 code suggestions across the 155,397 agent-review conversations, against 25,673 from human reviewers. Volume review is already delegated in practice; the open question for any team running Claude Code, Copilot, or CodeRabbit on its PRs is no longer whether agents participate in review but which checks they should own. The evidence available today supports a specific split: delegate first-pass triage, style, and minor quality checks to agents; keep security-critical review with humans backed by dedicated scanners; and refuse to cut reviewer hours until you have per-check catch rates from your own repos. The strongest reason for caution is a separate benchmark study in which Copilot’s reviewer, faced with hundreds of documented vulnerabilities, produced fewer than 20 comments, most about spelling or minor style.
What the new agentic review study actually measured
The study starts from a classification problem that matters for how much you trust its numbers. The authors identified 16 AI-based code review bots by manually checking each bot account name against official websites and documentation, deliberately separating AI agents from non-AI automation such as GitHub Actions, according to the paper’s method section. They also dated each agent’s capability: reviews count as AI agent reviews only from the agent’s capability announcement onward, and GitHub Copilot reviews are classified as agent reviews from May 19, 2025. That boundary is why this comparison is only now measurable. Agentic review on mainstream tooling is about eighteen months old as of this writing.
From that population, seven agents met the threshold of at least 383 review comments or code suggestions and appear across all three research questions: “CodeRabbit [5], GitHub Copilot [6], Gemini Code Assist [31], Cursor [32], Claude Code [33], Sourcery AI [34], and Greptile [35]” per the study. If your team runs any of these, the dataset plausibly includes review behavior like yours.
The quality method is the part worth stealing. The authors computed 111 code metrics, lines of code, executable statements, complexity, coupling, cohesion, and more, on source files before and after applying each of 8,985 adopted suggestions, using SciTools Understand, per the paper. That is the right instinct: judge a review suggestion by what it does to the code when accepted, not by how confident the comment sounds.
Who reviews how much, and why volume is not value
The headline split: “AI agents provide 88,011 code suggestions from a total of 155,397 conversations (56.6%),” the study reports, “while human reviewers provide 25,673 (20.8%) code suggestions.” Read those denominators carefully. The percentages are shares of conversations, and the counts are suggestions; the paper reports them exactly this way, and the gap between the two shares does not mean agents catch most defects. It means agents generate the most comments. Comment volume is a routing fact, not a quality fact.
Volume also hides an outcome gap the same paper measures directly. Code suggestions from AI agents were adopted into the codebase at a significantly lower rate than human suggestions, 16.6% versus 56.5%. In a statistically representative sample of unadopted agent suggestions, the most common reason for rejection was that the suggestion was incorrect or would break the code (28.7%), followed by developers applying an alternative fix (24.0%). And adoption is not free when it happens: adopted agent suggestions were associated with significantly larger increases in code complexity and code size than adopted human suggestions, across the study’s complexity and size metrics. The authors’ own summary is that agents can scale defect screening while human oversight remains critical for suggestion quality.
What humans add is measured too, not assumed. Over 95% of AI agent comments focus on Code Improvement or Defect Detection, while human reviewers provide a broader range: Understanding feedback (clarification questions about intent) accounts for 31% of human comments on human-written code, and Knowledge Transfer (pointing to conventions and existing code) for 4–6%, categories agents rarely supply. That is the positive case for keeping humans in the loop even where agents produce more comments.
There is a second reason to treat volume cautiously: agents produce comments cheaply and continuously, and humans ration attention. A production-scale characterization of GitHub Copilot found “the difference between quick agentic turnaround times and the minutes-long user idle periods at turn boundaries,” and built a lightweight idle-time predictor capturing 86–90% of total idle time. That finding is about coding sessions rather than review queues specifically, but the mechanism transfers: agents respond quickly, humans respond in minutes, and any workflow that parks a human reviewer in the first-pass path pays for that idle time. Delegating first-pass triage is how you reclaim it.
The cost side of the ledger has a forecast attached, not a measurement. One study of agentic coding economics cites a Gartner forecast that AI coding cost could exceed the average developer salary by 2028 under consumption-based pricing. That is a projection under stated assumptions, so treat it as a reason to track token spend on review agents, not as settled fact.
The security blind spot is documented, not hypothetical
Here is the adversarial evidence that should anchor your routing table. Across 7 benchmark datasets containing hundreds of documented vulnerabilities, SQL injection, insecure deserialization, cross-site scripting, GitHub Copilot’s Code Review “generated a total of fewer than 20 comments, most of which addressed spelling or minor style issues,” per Can AI Spot Security Flaws Before You Commit?. The authors’ conclusion is blunt: Copilot “is not ready to replace or even supplement dedicated security analysis tools.”
Two qualifications keep this honest. First, the study tested one tool, Copilot’s reviewer, on benchmark datasets rather than production PRs; it does not measure CodeRabbit, Greptile, or Claude Code on the same defects. Second, benchmarks are deliberately adversarial in a way real PRs are not. But the direction of the error is what matters for routing: when the agent missed, it missed the highest-severity defect class while still producing plausible-looking review output. A reviewer that comments on spelling in a file containing SQL injection is worse than a reviewer that says nothing, because it simulates coverage.
One clean review run proves nothing
The same Copilot study documents a failure mode that breaks naive evaluation: non-determinism. On the dvws-node project, Copilot reviewed 46 of 50 files with zero comments in the initial run, then generated 7 comments on re-review, some touching password input fields and sanitization, per the study. The vulnerabilities did not move; the reviewer changed its mind.
This has a direct operational consequence. If you evaluate an agent reviewer by running it once over a sample of PRs and counting catches, you are sampling from a distribution with one draw. The minimum viable check is a re-run protocol: run the agent at least twice over the same change set, measure comment overlap, and treat any check with high run-to-run variance as advisory only. It also means that a quiet week from your review agent is not evidence that the code was clean.
Stacking agent reviewers on agent authors is unproven
Many teams now have one agent write the code and a different agent review it. The coordination evidence suggests caution. In the CooperBench result cited by a study of agent cooperation, “agents were on average about 30% less successful when cooperating than when performing both tasks alone, with failures linked to poor communication, commitment deviations, and incorrect models of partner behavior.” That result comes from collaborative coding tasks, not code review workflows, so it does not prove reviewer stacks fail. It does mean the burden of proof sits on the team stacking agents: measure whether the second agent catches anything the first agent’s author loop and your human reviewers do not, before paying for the extra layer.
The review-conversation data sharpens that question without answering it. Agent-reviews-agent conversations are only 0.3% of the dataset, and 94.0% of them involve the same agent reviewing its own code, so genuine second-agent review is barely practiced, let alone measured. The same study’s cross-agent results suggest which agent you stack matters: Cursor concentrates almost exclusively on Defect Detection, 99% versus 40% overall for agents, with a large effect size, while Claude Code produces fewer Defect Detection comments than agents overall, 20% versus 40%, with a small effect size. The second agent’s focus, not just its existence, is part of what you would be measuring.
There is a configuration dimension too. Codifying your review standards into the agent’s harness depends on the tool reading a versioned context file from the repository, and the tools split on which file they read natively: “Native AGENTS.md support is already provided by Copilot, Cursor, and Codex; Claude Code, despite being the most popular tool, does not yet support it” per the harness-engineering study. Claude Code teams are not left with prompts and habit, though: CLAUDE.md and AGENTS.md “serve as de facto standards,” and “CLAUDE.md most frequently points to AGENTS.md (301 cases),” per the same study, so review standards can still live in versioned configuration under a tool-specific filename. That support matrix is as of that study and changes quickly, so verify your tool’s current support before building a routing policy on it. What the versioned file buys you is reproducibility: the same standards applied on every run, which makes per-check measurement easier to repeat.
The division-of-labor checklist
Combining the volume data, the adoption and outcome results, the security miss, and the non-determinism finding gives a routing table with explicit conditions. The split below is my inference from those sources; no cited paper proposes this exact checklist.
| Review check | Route to | Evidence and condition |
|---|---|---|
| First-pass triage, style, minor quality | Agent reviewer | Agents already supply the volume (88,011 vs 25,673 suggestions), but judge by adoption, not count: agent suggestions were adopted at 16.6% vs 56.5% for humans, and adopted agent suggestions raised complexity and code size more |
| Design intent, conventions, project knowledge | Human reviewer | Humans supply Understanding feedback on 31% of comments and Knowledge Transfer on 4–6%; agents rarely provide either |
| Security-sensitive diffs (auth, input handling, deserialization, queries) | Human reviewer plus dedicated scanner | Copilot’s reviewer missed hundreds of documented vulnerabilities while commenting on spelling |
| Anything where a miss is expensive and silent | Human sign-off required | Agent output is non-deterministic across re-runs, so absence of comments is not assurance |
| Review standards you want enforced uniformly | Agent, only if the tool reads a versioned context file | Native AGENTS.md support in Copilot, Cursor, and Codex; Claude Code lacked it as of that study but reads versioned CLAUDE.md, which the same study finds is a de facto standard alongside AGENTS.md |
| Agent-authored PRs reviewed by a second agent | Unproven; measure before trusting | Cooperating agents were ~30% less successful than solo agents on collaborative coding tasks, and in real review data 94.0% of agent-reviews-agent conversations are one agent reviewing its own code |
I would treat the first row as safe to adopt now, with two conditions attached: track the adoption and after-the-fact revert rates of agent suggestions, because suggestion counts flatter the agent, with 16.6% adoption and 28.7% of sampled unadopted suggestions judged incorrect, and watch complexity metrics on the suggestions you do accept. The security row is the one I would not negotiate on. The documented failure mode is exactly the class where a missed review becomes an incident, and the agent’s output in that study mimicked coverage.
What to measure before cutting reviewer hours
The bottleneck framing that follows from this evidence: review capacity stops tracking headcount and starts tracking routing. If agents absorb first-pass volume and humans concentrate on the diffs where agents demonstrably miss, the constraint becomes how fast you can classify and route a PR, not how many reviewers you employ. That is the upside case, and it requires measurement to claim.
Borrow the studies’ own instruments:
- Per-check catch rates. Split your review checklist into named checks (style, naming, error handling, injection surfaces, authz logic) and record who caught what, agent or human, per PR. Suggestion counts cannot answer this; only labeled catches can.
- Adoption and outcome. The 111-metric before/after method on 8,985 adopted suggestions is reproducible in miniature: pick the five metrics you care about (complexity, coupling, duplication) and compute deltas on accepted agent suggestions in your repos. The pattern to check for is the one the study found, adopted agent suggestions pushing complexity and size up faster than human ones.
- Rejection reasons. The study’s sampled breakdown of unadopted agent suggestions, 28.7% incorrect, 24.0% replaced by an alternative fix, 16.4% unnecessary or already handled, is a ready-made labeling scheme for your own triage data.
- Determinism. Re-run the agent on a sample of merged PRs and measure comment overlap, following the failure mode documented in the Copilot study.
- Idle time recovered. If first-pass delegation is working, human wait time at turn boundaries should fall; the production-scale Copilot characterization shows that idle time is measurable and largely predictable (86–90% captured by a lightweight predictor).
For evaluation discipline beyond these, Engineering Reliable Coding Agents offers the most structured starting point in this evidence set: a versioned catalog of 206 reliability records, including 193 gated practices with runnable evaluation and reliability protocols, synthesized from 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records. The gated-practice framing is the right posture: no review check moves from human to agent until its gate (measured catch rate on your code) passes.
Where this evidence runs out
Three limits deserve weight, not a ritual mention. First, the defect-class question in the title, who catches what, by defect type, is only partly answerable. The synergy study classifies feedback with the Bacchelli-Bird categories (Code Improvement, Defect Detection, Understanding, Knowledge Transfer), not CWE-level security classes. It can show that agents concentrate over 95% of comments on Code Improvement and Defect Detection while humans carry the intent-clarification and knowledge-transfer work, and it can rank agents against each other on feedback mix, but it cannot tell you whether CodeRabbit catches injection flaws better than Copilot did. Per-agent security detection at the vulnerability-class level remains measured only by the single Copilot benchmark study. Second, every quantitative anchor comes from single-study settings: benchmark repos for the security finding, one dataset of real conversations for the volume and adoption findings, a collaborative-coding benchmark for the cooperation penalty. Production repos with different review cultures could shift every number. Third, tool behavior dated fast even within this evidence: agentic Copilot review only exists as a classifiable category from May 19, 2025, and the AGENTS.md support matrix reflects one study’s snapshot.
The decision that survives those limits is narrower than the title’s question but more useful: delegate the checks where agent volume is real and a miss is cheap, keep humans and scanners on the checks where a documented miss is expensive, and treat any reduction in reviewer hours as an experiment gated on your own catch-rate data rather than on anyone’s suggestion counts.
Frequently Asked Questions
What is the adoption rate for AI agent code suggestions compared to human suggestions?
Code suggestions from AI agents were adopted into the codebase at a significantly lower rate than human suggestions, 16.6% versus 56.5%.
How did GitHub Copilot’s reviewer perform on security benchmarks?
Across 7 benchmark datasets containing hundreds of documented vulnerabilities, SQL injection, insecure deserialization, cross-site scripting, GitHub Copilot’s Code Review “generated a total of fewer than 20 comments, most of which addressed spelling or minor style issues,” per Can AI Spot Security Flaws Before You Commit?.
Which tools natively support AGENTS.md for review standards?
“Native AGENTS.md support is already provided by Copilot, Cursor, and Codex; Claude Code, despite being the most popular tool, does not yet support it” per the harness-engineering study.
Join the discussion
Share a useful perspective or ask a question about this article.