The AI Productivity Debate: Why Perceived Gains Are the Weakest Evidence Leaders Have

Why this debate matters now

Almost every organisation has already bought the tools. The open question is no longer whether to adopt AI assistance for professional work, but whether anyone can demonstrate what it returns — and to whom.

That question has become uncomfortable because the strongest-sounding evidence and the strongest evidence are not the same thing. Self-reported gains are large, immediate and easy to collect. Measured gains are smaller, slower, contested, and increasingly difficult to obtain at all. Boards, CFOs and engineering leaders are now allocating budget, headcount and roadmap capacity on the basis of the first category while quietly discovering that the second category refuses to cooperate.

This article is an opinion piece written for professionals who must make or defend those decisions: engineering and product leaders, transformation and operations executives, researchers studying technology adoption, and practitioners asked to justify tooling spend. It sets out the two strongest cases in the debate, identifies where the evidence genuinely agrees, and argues a position: the measurement question, not the capability question, is the decision-relevant one — and most organisations are still answering it badly.

Executive summary

• Well-conducted studies disagree sharply on the size and even the direction of AI's effect on professional productivity, and the disagreement is largely explained by task type, developer experience, codebase maturity and measurement method.
• The most reliable single finding in the literature is not a productivity number at all. It is a calibration failure: in a controlled trial, experienced developers' perception of their own speed diverged from measured time by roughly 40 percentage points.
• Large multi-firm field experiments show meaningful positive effects on task throughput; small high-context trials in mature codebases have shown neutral or negative effects. Both can be true simultaneously.
• Measured speed gains systematically overstate value gains, because cheap tasks get done that would previously not have been worth doing.
• Quality and stability signals have moved in the opposite direction to throughput signals. Security pass rates for AI-generated code have been effectively flat for two years while functional correctness improved sharply.
• Rigorous randomised measurement is becoming harder as adoption deepens: participants increasingly refuse to work without AI, which biases experiments and removes exactly the evidence leaders most need.
• Macroeconomic evidence shows a familiar pattern — perceived firm-level gains exceeding measured ones, consistent with historical technology adoption lags rather than with either boosterism or dismissal.
• The practical implication for professionals is a measurement discipline, not a verdict: instrument outcomes, not sentiment; measure downstream cost, not just upstream speed.

What is actually being disputed

Most "AI productivity" arguments collapse three distinct claims into one, and the participants then talk past each other. Separating them is the precondition for a useful debate.

Claim A — Speed. AI reduces the time required to complete a given task.
Claim B — Throughput. AI increases the number of units of work completed per person per period.
Claim C — Value. AI increases the economic worth of what an organisation produces.

These are not interchangeable. A tool can raise A and B while leaving C flat, if the additional units are low-value work that AI made cheap enough to attempt. METR's research team has argued explicitly that speed measures are likely biased upward relative to value measures, precisely because of this task-substitution effect.Speed measures may be inflated by individuals doing additional tasks which AI can do well or quickly but which would otherwise not have been worth prioritising metr

Callout — The substitution trap
If a task previously cost eight hours and now costs twenty minutes, it will get done. That is a genuine speed gain. Whether it was worth doing at either price is a separate question that no stopwatch can answer.

The case that AI is delivering real gains

The strongest version of the optimistic argument rests on field experiments rather than vendor claims.

The largest of these pooled three randomised trials run inside Microsoft, Accenture and an unnamed Fortune 100 company as ordinary business operations. Across 4,867 developers, the combined analysis found a 26.08% increase in completed tasks among those given access to the AI coding assistant, with less experienced developers showing both higher adoption and larger gains. The result is published in a peer-reviewed management journal, which places it above most material circulating in this debate. INFORMS

Enterprise-scale internal research points the same way. A Google randomised trial with 96 full-time engineers on a realistic C++ task found that developers using AI were roughly 21% faster after controlling for other factors, though the confidence interval was wide. In practical terms, the AI group averaged 96 minutes against 114 minutes for the control group. An earlier controlled study of GitHub Copilot on a self-contained JavaScript task reported a much larger effect, around 56% faster completion, with participants underestimating rather than overestimating the benefit. arXiv + 2

Organisational survey evidence has also shifted. Google's DORA programme, drawing on responses from nearly 5,000 technology professionals and more than 100 hours of qualitative data, found that AI's relationship with software delivery throughput reversed between 2024 and 2025, moving from negative to positive, alongside improvements in product performance and time spent on valuable work. Adoption is now close to universal: 90% of respondents reported using AI in daily work, a 14% increase on the prior year, with a median of about two hours per day. Google Cloud + 2

The optimist's summary: multiple independent designs, in different organisations, on different task types, find positive effects. That is not nothing.

The case that the gains are overstated

The sceptical argument does not deny those results. It disputes their generalisation — and it has one exceptionally awkward data point.

In a randomised controlled trial conducted between February and June 2025, METR recruited experienced open-source contributors to work on real issues in repositories they already knew well, randomising each task to allow or disallow AI. Sixteen developers worked across 246 tasks in large, mature repositories, using tools that were frontier-grade at the time, and took 19% longer with AI than without. The headline number matters less than the calibration gap around it: participants forecast a 24% speedup beforehand and still reported a 20% speedup afterwards, against a measured 19% slowdown. METRarxiv

Quality signals compound the concern. Veracode's longitudinal benchmark, updated in spring 2026, reports that only about 55% of generation tasks produce secure code, meaning a known security flaw is introduced in roughly 45% of cases — a rate that has stayed close to flat for two years even as syntax correctness exceeded 95%. Commit-level analysis from the code-analytics vendor GitClear, covering more than 600 million analysed changes from 2023 to 2026, reports refactoring line moves down 70%, cross-file function calls down 35%, code-block duplication up 81%, copy/paste up 41% and two-week churn up 15%. VeracodeGitClear

DORA's own data carries the same warning from a different direction: while throughput improved, AI adoption continued to be associated with increased delivery instability, suggesting teams adapted for speed faster than their systems adapted to absorb it. LCMH

Callout — Read vendor research carefully, not dismissively
GitClear and Veracode both sell products adjacent to the problems they document. That is a reason to check methodology, not a reason to discard findings that are longitudinal, large-sample and consistent with independent survey data. The same standard should be applied to vendor research showing gains.

Where the debate is genuinely settled

Stripping out the rhetoric, several propositions now command broad support across studies that otherwise disagree.

The measurement problem is now the story

Here is the development that should worry professionals most, and it has received far less attention than any headline percentage.

On 24 February 2026, METR published an update explaining that its follow-up experiment — 57 developers across 143 repositories and more than 800 tasks, begun in August 2025 — had produced an unreliable signal. The reason was adoption itself. An increasing share of developers declined to participate because they did not want to work without AI, and 30% to 50% of participants said they were choosing not to submit certain tasks because they did not want to attempt them unaided. The raw figures showed an estimated 18% speedup among returning developers (confidence interval −38% to +9%) and 4% among newly recruited developers (−15% to +9%), but the team concluded that these were likely a lower bound and that the study design itself needed changing. Their considered judgement was that developers are "more sped up from AI tools now" than a year earlier, while stating plainly that their data was only very weak evidence for the size of that increase. metr + 4

This is the crux of the debate, and it is genuinely uncomfortable for both camps. The rigorous method is degrading precisely because the technology is being adopted. Refusal to work without a tool is itself evidence of perceived value — but it is not measurement, and it cannot be converted into one.

Figure 1 — proposed diagram for the OneWise design team

Title: The Evidence Ladder for AI Productivity Claims
Purpose: Show readers how to rank the credibility of any AI productivity claim they encounter.
Format: Vertical ladder, five rungs, ascending. Colour gradient from muted grey (bottom) to deep teal (top). Left-hand vertical axis labelled "Confidence"; right-hand annotations labelled "Common failure mode".
Rungs, bottom to top: (1) Vendor case study — failure mode: selection of successes; (2) Self-reported survey — failure mode: calibration error, ~40-point gap documented; (3) Observational telemetry — failure mode: confounding, no counterfactual; (4) Field experiment / quasi-experiment — failure mode: limited external validity; (5) Randomised controlled trial — failure mode: participation selection effects, increasing over time.
Additional element: A downward red arrow beside rung 5 labelled "Reliability declining as adoption rises (METR, Feb 2026)".
Caption: Higher rungs are more credible, but the top rung is becoming harder to reach.

Latest developments

Dated, so readers can assess currency.

Two of these deserve elaboration. The macroeconomic picture, from a survey of nearly 750 corporate executives, found positive labour-productivity gains concentrated in high-skill services and finance, driven by revenue-based total factor productivity rather than capital deepening — alongside an explicit productivity paradox in which perceived gains exceed measured gains, likely reflecting delayed revenue realisation. That is the classic signature of a general-purpose technology mid-diffusion, familiar from Paul David's analysis of electrification. NBER

The METR survey is the more interesting counterweight to METR's own RCT. Among 349 technical workers surveyed between February and April 2026, median self-reported value uplift ranged from 1.4x to 2x, while the median self-reported speed change was 3x — respondents retrospectively estimated 1.3x value in March 2025, 2x in March 2026 and forecast 2.5x for March 2027. The authors' handling of their own result is a model of intellectual honesty: METR staff reported the lowest value gains of any subgroup, and where researchers could inspect public outputs behind the very highest self-reported multipliers, they judged those claims overstated. metrmetr

What good measurement looks like in practice

If the debate cannot be settled globally, it can be settled locally — inside your own organisation, for your own task mix. That requires abandoning the single-number question.

Figure 2 — proposed diagram for the OneWise design team

Title: From Speed to Value: A Four-Gate Measurement Pipeline
Purpose: Give leaders a defensible sequence for evaluating AI tooling, replacing single-metric ROI claims.
Format: Horizontal flow, four gates left to right, each a rounded rectangle with an input arrow above and a "leakage" arrow below.
Gates: (1) Speed — inputs: task start/stop instrumentation; leakage: review time excluded. (2) Throughput — inputs: merged units of work; leakage: unit size inflation. (3) Quality — inputs: change failure rate, defect escape, security findings, churn; leakage: deferred defects. (4) Value — inputs: outcome metrics tied to product or customer; leakage: task substitution into low-value work.
Overlay: A dashed feedback loop from Gate 4 back to Gate 1 labelled "Recalibrate task selection".
Caption: Gains measured at Gate 1 routinely fail to survive to Gate 4. Measure where the value is claimed, not where it is easiest to observe.

Three practices distinguish organisations that can answer the question from those that cannot. First, pair every velocity metric with a stability metric and report them together, never separately. Second, measure unit size, because a 50% increase in merged pull requests means something different if pull requests grew by half. Third, fix the denominator before the tool arrives — retrofitted baselines are the single most common source of unfalsifiable ROI claims.

Common myths worth retiring

• "The RCT proved AI makes developers slower." It measured one setting — experienced contributors, mature repositories, early-2025 tools — and the authors themselves now describe the finding as historical.
• "95% of AI pilots fail." The figure comes from a working paper based on 52 interviews and 153 leaders, has been criticised for a narrow six-month P&L definition of success and for a conflict of interest, and does not support the sweeping use it receives.
• "Developers know whether they are faster." The best available within-study evidence says they do not.
• "Benchmarks tell us what to expect at work." Benchmark performance and deployed productivity have diverged repeatedly; security is the clearest example, where functional scores rose while security pass rates did not.
• "AI will fix a struggling team." DORA's central finding runs the other way: AI amplifies whatever capability already exists.

Practical takeaways by role

Engineering and platform leaders. Instrument change failure rate and review latency before expanding licences. Treat a rise in throughput without a rise in stability investment as a leading indicator of future incident load. Require security scanning on AI-authored paths as policy, given a generation failure rate that has not improved with model quality.

Executives and finance. Discount self-reported multipliers heavily and demand the denominator. Ask which of speed, throughput or value the claimed number measures — the answer is usually speed, and value is what appears in the accounts. Budget for the complementary investment (process redesign, review capacity, data quality) that the macro evidence identifies as the actual bottleneck.

Individual professionals. Assume your perception of your own speedup is unreliable in both directions, and keep a small sample of timed tasks. Notice where AI shifts what you work on, not just how fast — that shift is where value quietly leaks.

Researchers and analysts. Report value and speed separately, disclose selection effects, and treat participation refusal as data rather than attrition.

Key insights

1. Speed, throughput and value are three different claims; conflating them causes most of the disagreement in this debate.
2. The single most robust finding is a calibration failure, not a productivity number.
3. Effect sizes are strongly task-dependent — constrained tasks favour AI, high-context work in mature systems does not, at least in the studies conducted so far.
4. Less experienced practitioners consistently show larger measured speed gains.
5. Throughput and stability have moved in opposite directions, and reporting one without the other is misleading.
6. Security pass rates for generated code have stayed roughly flat for two years while functional correctness improved sharply.
7. Randomised measurement is degrading as adoption rises, because participants increasingly refuse the control condition.
8. Refusal to work without a tool is evidence of perceived value but is not a measurement of it.
9. Firm-level perceived gains exceed measured gains — the classic mid-diffusion pattern, not proof of either hype or failure.
10. The organisations that will answer this question first are those that fixed their baseline before the tools arrived.

Frequently asked questions

Do AI coding tools make developers faster?
Sometimes substantially, sometimes not at all. Field experiments on enterprise and constrained tasks report gains in the 20–55% range, while one controlled trial in mature open-source repositories measured a 19% slowdown. Task type, codebase maturity and experience level explain most of the variation.

Which is the most reliable study?
None individually. The peer-reviewed multi-firm experiment covering 4,867 developers has the largest sample; the METR trial has the most realistic task selection; both have known limitations. Treat the pattern across studies as the evidence, not any single headline.

Why did METR change its experimental design?
Because adoption broke the control condition. Developers increasingly declined to participate or withheld tasks they did not want to attempt without AI, biasing the estimate downward.

Is the "19% slower" figure still current?
No. It describes tools available between February and June 2025 in one setting, and METR now labels the result historical while judging that developers are likely more sped up today.

Is the "95% of AI pilots fail" statistic reliable?
It should be used with heavy caveats. It comes from a working paper with a small sample, a narrow six-month definition of success, and criticism regarding the authors' commercial alignment.

Why do perceived gains exceed measured gains?
Two mechanisms. The visible generation step is memorable while verification is not, and speed measures capture cheap new tasks that add less value than time saved implies.

Does AI improve or degrade code quality?
Survey respondents report improvement; independent commit-level and security analyses report degradation on several dimensions. These measure different things, and both should be reported.

Should we stop using AI tools until the evidence resolves?
That does not follow from any of the evidence. The defensible response is to measure locally and pair velocity with stability, not to abstain.

How do we build a credible internal baseline?
Capture cycle time, change failure rate, review latency and unit size for at least one quarter before a rollout, and preserve a comparison cohort where feasible.

What is the single most misleading metric?
Self-reported time savings, because it is the easiest to collect and the least accurate.

Do juniors or seniors benefit more?
Measured speed gains are consistently larger for less experienced practitioners. Whether this affects long-term skill formation is an open research question.

Does AI reduce burnout?
DORA's 2025 data found no measurable effect on friction or burnout, characterising both as properties of the organisational system rather than the individual toolset.

What does the macroeconomic evidence show?
Positive but modest measured gains concentrated in high-skill services and finance, with perceived gains running ahead of measured ones.

Is this just the Solow paradox again?
It resembles it closely. Paul David's work on electrification showed that measured returns can lag adoption by years while complementary reorganisation catches up.

What would change your mind about the sceptical case?
Sustained improvement in stability and security metrics alongside throughput gains, and randomised or quasi-experimental value measurement at organisational level.

Glossary

Calibration error — The gap between a person's estimate of a quantity and its measured value.
Change failure rate — The proportion of deployments that cause a degraded service requiring remediation.
Code churn — Lines modified or deleted shortly after being written, often within two weeks.
Confidence interval (CI) — The range within which a true effect is likely to lie; wide intervals indicate low precision.
DORA — Google's DevOps Research and Assessment programme, which publishes annual research on software delivery performance.
External validity — The extent to which a study's result generalises beyond its specific setting.
METR — Model Evaluation and Threat Research, a nonprofit that measures AI system capabilities and impact.
Randomised controlled trial (RCT) — A design that randomly assigns units to treatment and control conditions to isolate causal effect.
Selection effect — Bias introduced when participation or inclusion is systematically related to the outcome studied.
SPACE framework — A multidimensional model of developer productivity spanning satisfaction, performance, activity, communication and efficiency.
Solow paradox — The observation that technology adoption can appear everywhere except in productivity statistics.
Task substitution — A shift in which tasks are attempted because a tool changes their relative cost.
Total factor productivity (TFP) — Output growth not explained by increases in labour and capital inputs.
Value stream management — Practices that connect individual work to end-to-end delivery of customer value.

References

Academic papers and preprints

Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. METR. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2026). The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Management Science. https://doi.org/10.1287/mnsc.2025.00535

David, P. A. (1990). The dynamo and the computer: An historical perspective on the modern productivity paradox. American Economic Review, 80(2), 355–361.

Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021). The SPACE of developer productivity. ACM Queue, 19(1), 20–48.

Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024). How much does AI impact development speed? An enterprise-based randomized controlled trial (arXiv:2410.12944). https://arxiv.org/abs/2410.12944

Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot (arXiv:2302.06590). https://arxiv.org/abs/2302.06590

Government and central bank sources

Baslandze, S., Edwards, Z., Graham, J., McClure, T., Meyer, B. H., Sparks, M., Waddell, S. R., & Weitz, D. (2026). Artificial intelligence, productivity, and the workforce: Evidence from corporate executives (NBER Working Paper No. 34984). National Bureau of Economic Research. https://doi.org/10.3386/w34984

Industry and institutional research

Becker, J. (2026, May 11). Measuring the self-reported impact of early-2026 AI on technical worker productivity. METR. https://metr.org/blog/2026-05-11-ai-usage-survey/

Becker, J., Rush, N., Cunningham, T., Rein, D., & Mahamud, K. (2026, February 24). We are changing our developer productivity experiment design. METR. https://metr.org/blog/2026-02-24-uplift-update/

DORA / Google Cloud. (2025). State of AI-assisted software development. https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report

GitClear. (2026). The maintainability gap: 2026 AI code quality research. https://www.gitclear.com/the_ai_code_quality_maintainability_gap

Veracode. (2026). Spring 2026 GenAI code security update. https://www.veracode.com/blog/spring-2026-genai-code-security/

Note on sourcing: GitClear and Veracode are commercial vendors publishing their own research; their findings are reported here as vendor longitudinal analyses and should be weighed accordingly. The "GenAI Divide" working paper referenced in the myths section is discussed as a contested source and is deliberately not cited as evidence.