METR already redid the study at a later date and now finds a likely 18% speedup
"For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post)
I would not, at all, suggest that this second study corrects or debunks the first.
Instead what it shows (if anything, i.e. if you can even put aside the regrettable choice to change the payment level, which affects applicant recruitment) is that the mindset shift has already happened: developers now don’t want to attempt some tasks without AI.
What that tells you is not (with any confidence at least) that they are faster, but perhaps that we are beyond the point that this can be meaningfully measured. AI could still be making developers slower, but developers aren’t going to be willing or perhaps able to help you find out.
Basically the job is different now.
What this does for me, perhaps, is vindicate my feelings. I can do agentic coding; I have learned the principles and some tools and I could learn more. But if this study is really reflective of how other developers feel now, I am done.
(Or that the failure of the second study validates the conclusions of the first.)
I am just saying that people here who think the second study overturned, debunked or corrected the findings of the first are explicitly wrong, because even its authors admit it is a broken study.
It would take a non-broken study to do that, and it may not actually be possible anymore, which is perhaps the most useful finding of the second study.
I do think AI has been a huge boon to productivity in many ways, but looking at feature timelines, I think it's pretty clear the 'critical shortest path' of key features hasn't been sped up by that much.
I think there is a simple reason for that. If you automate something, you make the measureable/predictable thing faster. So the hard to measure/predict part of the job will take more share of the time, and overall difficulty to measure/predict goes up.
I think this is what happened with Agile Scrum - as developers became more productive (for unrelated reasons, two main sources of SW developer productivity before AI were compilers and open source), the bureacracy (amount of meetings) increased, because the ratio of hard to measure vs easy to measure went up. Bureacracy is hard to measure, so it went up (as a share of work). I expect this only getting worse with more automation, such as AI. So I predict an increase in share of bureacracy compared to pre-AI world.
Either way, IMHO main point is automation has the opposite effect on human job predictability, it lowers it. Tasks we can easily automate are those that are easy to predict.
Overall this suggests to them that the current speedup is likely greater than what the study could measure.
Like, what people are saying is, “That old study was wrong! They did a new broken study that overturned it!”
Which is ancient at this point, and half a year older than the November 2025 inflection point when agentic coding got really good.
The original article is from August 2025, and the overall message to not trust ‘how it feels’ and rather measure outcomes seems right to me despite the outdated figures. On my team at least, we are seeing a noticeable inflection in work shipped with AI according to Weave.