Hacker News Reader

Comment by kalkin | Hacker News Reader

> Meanwhile, a study (late 2025) seems to report that although participating developers felt they completed tasks faster using AI, they where around 19% slower.

METR reran the study early this year and, while they caveat it, this time they found a speedup, which is consistent with subjective estimates of productivity also having increased -- the simplest explanation is that subjective estimates exaggerate, but there's still a speedup with current models: https://metr.org/blog/2026-02-24-uplift-update/#wider-adopti...

(Nobody seems to cite the followup since it's not such a fun counterintuitive finding.)

View on HN

"Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools."

That's more than a caveat. That says, this study is unreliable. And why? "The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI."

So, maybe people are fooling themselves, just like this article suggests (I almost wrote asserts, but saying, "I think..." isn't really an assertion)

> The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI.

Why do you think this is? Who wants to spend an afternoon gluing API calls together when the robot can do it in 5 minutes?

If you look at the data 25% of the people who reported attempting a task without AI did not complete it. Seeing as the time spent on AI tasks was not that much higher than the time spent on completed Non-AI tasks, I suspect it was less about unwillingness, and more about inability.

We should really be studying AI related cognitive and/or skill decline more heavily.

Don't let the downvotes get you down. They don't know who funds or constitutes METR. If you actually read these papers, you'll find they cite themselves 4-5 times each publication. They also consider "hosting an HTTP server using Python" to be a "long time horizon" task.

The OP cited a METR study... if you don't think they're credible, fine, but it's hard for me to see how it could be intellectually honest to cite them when their research is helpful to your argument and discredit them when it's not. One might even call this approach "fooling yourself".

The OP is not citing their claims, but their commentary. Do you understand why I find that to be different than citing their findings? We're not talking percentages here.

The people who chose to use AI would be the ones who would be fooling themselves, right? But they measured actually positive productivity there.

METR's concern makes most sense if it's about the inability to measure a consistent trend over time, not about the validity of the study for the population that participated.

Have you looked at the raw data of the follow-up? It doesn't paint a pretty picture, many tasks attempted but not completed, refusal to attempt without AI, higher overall estimates and completion times than the previous study (both for AI and Non AI).

There's no definitive conclusion about productivity that can be drawn from the followup. However I think it's reasonable to argue that AI use overtime has made devs worse without AI.

Maybe "using AI" is not sufficient by itself. Maybe you need to use it in the right way/setup/environment.

By now, anyone writing articles in 2026 but ignoring the 2026 study is either intentionally trying to mislead people, or so uninformed on the topic that they're just regurgitating old anti-AI content.

If you want to talk about studies, we need to be talking about current studies with current tools. Not studies about how the situation was in the past with different tools.

> intentionally trying to mislead people

Is a bit hard. But I agree that the field is moving very fast and that it is hard to find studies that are up to date on the latest tooling.

Reply on news.ycombinator.com