Preprint: Fable 5 did not significantly raise pull request merge rates
According to a Harvard researcher, access to Fable 5 raised the 7-day merge rate of GitHub pull requests by 1.4 percentage points, a difference that was not statistically significant. Fable 5 scored 10.8 points higher than Opus 4.8 on SWE-Bench Pro.
Yining Hua, a researcher at Harvard University, published the study as an arXiv preprint on 17 November. June 2026 provided a natural experiment: a US export control order cut off access to Claude Fable 5 for 19 days (12 June–1 July). Most users switched to Opus 4.8.
Hua tracked pull requests, or proposed code changes, on GitHub. An accepted change is merged into the project’s code. The metric was the share of pull requests merged within seven days. Hua identified the model from the pull request description when the developer had left in the line that the coding agent Claude Code adds to commits. Only about 1 per cent of merged Claude Code pull requests carried such a line, and the share fell during the suspension. When Hua repeated the analysis on all pull requests from the same developers, the result pointed the same way.
The study covered 297 Fable 5 users. The 8,908 developers in the control group used Opus 4.8 between 26 May and 15 July but never Fable 5. The difference-in-differences method compares the change among users with the change in the control group over the same days.
While Fable 5 was available, its users had a merge rate 1.4 percentage points higher than during the suspension (95 per cent confidence interval −1.4 to +4.3). The difference was not statistically significant. That does not show there was no effect, however. In Hua’s interpretation, Fable 5 raised the merge rate by at most 4.3 percentage points. The short suspension leaves the confidence interval wide.
Hua also examined upgrades to the Opus and Sonnet models. Across five upgrades, the merge rate of developers who switched models changed by an average of −0.9 percentage points (−1.9 to +0.2), which was not statistically significant. In the six upgrades for which SWE-Bench Pro scores were available for both models, the new model scored 5 to 18 points higher, but the merge rate did not change significantly in any of them. Developers chose their own models, so the upgrade comparison shows only an association.
The metric has limits. In the sample, the authors themselves merged 95 per cent of pull requests. Hua notes that a merge shows only that the developer considered the work finished; it says nothing about code quality. If differences between models show up mainly when another person reviews the code, the estimates are too low. Almost all pull requests were already being merged before the upgrades, so the share had little room to rise.
Hua suggests that developers give agents tasks that an older model can also handle. SWE-Bench Pro accepts a solution when it passes the tests of the original fix. The benchmark does not measure, for example, how a model interprets a vague request. Buyers who judge a model’s strength by its scores are relying on a difference that likely does not show up in productivity, Hua writes.
The data and code are in the GitHub repository ningkko/benchmarks-vs-merges.
Read more: How an agent works · Agent map