The Two Results
Study A · GitHub Copilot, May–June 2022
Run by: researchers at GitHub and Microsoft Research (Peng, Kalliamvakou, Cihon & Demirer, 2023).
Participants: 95 professional programmers recruited as freelancers on Upwork, randomly split into a Copilot group (45) and a control group (50). Most were 25–34 years old and based in India or Pakistan, with about 6 years of coding experience on average.
Tool: GitHub Copilot, then powered by OpenAI's Codex model, just before its general release. The Copilot group watched a 1-minute introductory video. The control group could not use Copilot but could use web search and Stack Overflow.
Task: Implement a basic HTTP server in JavaScript from the same written instructions, with a skeleton repository and 12 tests the participants could see but not change.
Pay: Compensation depended on time to completion, so participants were rewarded for speed.
Outcomes: Completion time, from creating the repository to the first commit passing all 12 tests; and task success.
Result: Among participants who completed the task, the Copilot group averaged 71 minutes and the control group 161 minutes: 55.8% less time (95% CI 21% to 89% less). The Copilot group's success rate was 7 percentage points higher, a difference that was not statistically significant. Afterward, participants estimated that Copilot made them about 35% faster, an underestimate of the measured effect. The study did not measure code quality.
Study B · METR, February–June 2025
Run by: METR, an independent AI research organization (Becker, Rush, Barnes & Rein, 2025).
Participants: 16 experienced open-source developers who regularly contribute to large, mature repositories (on average 23,000 stars and 1.1 million lines of code). They averaged about five years and 1,500 commits on their repository. Most had used tools like ChatGPT; 44% had used Cursor before.
Tasks: 246 real issues (bug fixes, features, and refactors) that developers listed from their own repositories, about 2 hours each on average. Developers forecast each issue's time, then each issue was randomly assigned to allow or disallow AI: 136 allowed, 110 disallowed.
Tools: When AI was allowed, developers could use any tool; they mostly used Cursor Pro with Claude 3.5 or 3.7 Sonnet, the frontier models at the time. When AI was disallowed, no generative AI was permitted.
Pay: $150 per hour.
Outcome: Total implementation time, including time after pull-request review, reported by the developers and checked against screen recordings. Pull requests went through each repository's normal review.
Result: Allowing AI increased completion time by an estimated 19% (95% CI 2% to 39% slower). Before the tasks, developers forecast a 24% speedup; afterward, they still believed AI had sped them up by 20%. Economics experts had forecast 39% faster and ML experts 38% faster.
Group Task · 5 Minutes
Work in groups of three or four.
- Read · 2 minutes: inspect the two study cards.
- Compare and write · 3 minutes: identify who, what task, what treatment and baseline, and what outcome. Write one sentence reconciling the results. Name populations, tasks, or conditions; “it depends” alone is not enough.
Our reconciliation:
If you finish early: choose an adoption decision and sketch the evaluation that would change it. Name your target users and tasks, comparator, primary outcome, and one quality guardrail.
The adoption decision, outcome, and quality guardrail we would test:
Optional Full Comparison
Use this table to deepen the comparison after the short task. Short phrases are enough.
| Question | Study A | Study B |
|---|---|---|
| Who? | ||
| What task? | ||
| What baseline? | ||
| What outcome? | ||
| What was randomized? | ||
| What can generalize? | ||
| Biggest limitation? |
Optional Design Challenge
Suppose you lead an engineering organization in 2026. Sketch a third evaluation that would be more useful for deciding whether and how your teams should adopt coding agents.
Specify:
- target developers and repositories;
- assignment unit and baseline;
- primary outcome and one quality guardrail;
- how you will measure work when several agents run in parallel; and
- the result that would change your adoption decision.
Optional Follow-Up: A Later Complication
In February 2026, METR reported that a follow-up study no longer provided a reliable estimate of current productivity effects. Developers who valued AI were less willing to enter or remain in a study that prohibited it for half their work; developers selectively omitted tasks they particularly wanted AI to handle; parallel agents complicated time accounting; and task quality could differ between conditions.
Discuss:
- Which of these problems is selection bias?
- Which is a measurement problem?
- Could “time to finish the assigned task” become the wrong outcome as tools change what work people choose to attempt?
- What is exemplary about declining to promote a cleaner headline from unreliable data?
Sources
Original studies:
- Peng, Kalliamvakou, Cihon & Demirer (2023), “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”, arXiv:2302.06590
- Becker, Rush, Barnes & Rein (2025), “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, arXiv:2507.09089
Summaries and follow-up: