Robert Stowe

Study of 567 Claude Code pull requests: 84% were merged

A Welcome to Utah highway sign in front of a desert valley and low mountains, under a sky turning pink at dusk
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this paper, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open-source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs are integrated without further modification. The remaining 45.1% require additional changes benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.
arxiv.org

A study on arXiv looked at 567 pull requests opened with Claude Code across 157 open-source projects. 83.8% were accepted and merged. Of the merged ones, 54.9% went in without modification and 45.1% needed a person to change something first, most often for bug fixes, documentation, and adherence to the project's own standards. The agent was used first for refactoring, documentation, and tests.

I think the useful reading is not the headline acceptance rate. It is the combination: the tasks people chose to hand to the agent, and the share of accepted work that still needed a human pass.

The tasks tell you where to start. Refactoring, documentation, and tests are where projects already trusted the agent, and I think they are where a team should expect the first real wins. The 45% tells you what to budget. Nearly half of the accepted changes needed someone to finish them, for the things an outsider misses: the project's conventions, the edge of a bug fix.

My view is that an adoption plan built on those numbers looks different from one built on the slogan. It points the agent at the tasks it is already good at and keeps review time in the schedule.

What was the first kind of work your team was willing to hand to an agent, and did it match what the study found?

Photo source: https://photos.robertstowe.com/utah