METR AI Study of Developer Productivity Loss with AI
I so disagree with it. And then I don't.
Time after time an AI study makes the round.
“Against Expert Forecasts and Developer Self-Reports, Early 2025 Ai Slows Down Experienced Open-Source Developers”
Wow, totally not my experience. My experience with Go/Claude Code has been great. I’m much more productive than before - measured buy I can create several features a day where before it would have taken one day to create one of them.
How comes?
Looking deeper at that study it makes a lot of sense and surfaces the common problems with AI.
Overoptimism. Developers are generally the most overoptimistic crowd I’ve met. Especially in what they can do. No wonder developers are overoptimistic with AI. Which - as usual - needs some years of contrary experience for people to adapt to reality.
AI does not work well with large code bases. The study uses code bases with >1M LOC 10 years old. Code bases need to be AI friendly, not human friendly. This means code that matches context sizes, e.g. modularization code fitting context windows, tools and prompts that acknowledge size.
Developers do not well without training. Most prompts are bad. Especially command prompts vs. plan prompting. Inexperienced developers will not be able to write the right prompts.
Tacit knowledge - AI does not work well with implicit knowledge. AI works better with frameworks that use coding rules and languages that er explicit - it works worse with "convention over configuration".
Tacit knowledge II - Organizations do not have all knowledge, documentation and requirements as documents for the AI to read. Most of this is in the heads of developers. Migrating to AI means getting more knowledge into documents (whys, business context, architecture, past problems etc.)
Delegation is a general problem (“Developers accept <44% of AI generations). The biggest problem people have is "This is not what I wanted" Delegation means to learn to accept a solution that solves your problem and not compare a solution to your solution. AI is a delegation problem.
But there are some criticisms I have nevertheless:
Open Source Developers. The study is about open source developers - or more important open source projects - but then talks about developers mostly. If you’re a open source project, this is highly relevant to you. If not, it’s not. Open Source projects (having worked on some myself) are very different from e.g. startups.
I started wanting to disagree with the study, but in the end I would agree with it!
Who knew!