AI Aulis

AI NEWS IN ENGLISH · 19 November 2026

The most important AI news every morning. AI writes and checks the stories against the original sources – check important facts in the sources.

38 stories · next issue tomorrow morning · How the paper is made →

← Issue 19 November 2026Paper

Preprint: verdict on Devin’s code survival depended on how it was measured

AI-generated contentWritten by AI

arXiv · Source published: 16 November 2026 at 21:09 (UTC+02:00)

Researchers at Concordia University ran 88 analyses on the same dataset of 854,139 lines of code. The result for human-supervised coding agents pointed the same way throughout, but the result for Devin’s code varied with the measurement choices.

Concordia University researchers Musfiqur Rahman, Mahi Begoug and Emad Shihab published a preprint on arXiv on 16 November. They revisited the data from their earlier study: 2,802 merged pull requests that added a total of 854,139 lines of code to 215 repositories in the AIDev dataset. The researchers measured how long a line survives unchanged after it is merged.

Earlier studies have reported that code from coding agents survives longer than human code, for as long, or for less time. The researchers varied nine measurement choices, such as how lines are tracked and whether code is compared across repositories or within the same repository.

The tracking method alone changed the figures considerably. When the same sample lines were tracked by their original line number, 64.5 per cent of human-written lines appeared to have changed. When a line was also identified after it moved or its file was renamed, the share was 29.0 per cent. In a manually checked sample, only 35.9 per cent of the modifications recorded by line-number tracking were real.

For Devin, a fully autonomous agent, the measurement choices decided the result. When code was compared across repositories and each line was treated as a separate observation, 15 of 22 analyses found that Devin’s code was modified sooner than human code. Lines in the same repository, however, are not independent observations. When the researchers accounted for this in estimating statistical uncertainty, three such results remained, no more than randomly shuffled agent labels produced. Within repositories, no analysis found a difference. Devin was the only agent in the fully autonomous group, and its code appeared in 31 repositories. The researchers therefore said the effect of the Devin tool cannot be separated from that of autonomous operation.

The human-supervised agents included GitHub Copilot, OpenAI Codex, Claude Code and Cursor, with Copilot accounting for the largest share of the group’s lines. Their code survived longer than human code in more analyses than chance would explain. The measurement choices changed the size of the difference but not its direction. Humans made 87.7 per cent of the modifications to agent code.

The researchers re-ran the designs of five published studies on their own data. Each design’s result changed when a single choice was altered. Whether an agent’s code outlives human code depends as much on the measurement method as on the code itself, the researchers said. The comparison is observational and covers only merged pull requests. Agents’ pull requests are rejected more often, so the agent code studied is a selected sample. The lines were added between late December 2024 and late July 2025, and changes were tracked until the end of 2025, so every line was followed for at least five months. The researchers said that is a short time for code that may be in use for years.

Report an error in this story