Hacker News Reader

Comment by krupan | Hacker News Reader

"Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools."

That's more than a caveat. That says, this study is unreliable. And why? "The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI."

So, maybe people are fooling themselves, just like this article suggests (I almost wrote asserts, but saying, "I think..." isn't really an assertion)

View on HN

The people who chose to use AI would be the ones who would be fooling themselves, right? But they measured actually positive productivity there.

METR's concern makes most sense if it's about the inability to measure a consistent trend over time, not about the validity of the study for the population that participated.

> The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI.

Why do you think this is? Who wants to spend an afternoon gluing API calls together when the robot can do it in 5 minutes?

If you look at the data 25% of the people who reported attempting a task without AI did not complete it. Seeing as the time spent on AI tasks was not that much higher than the time spent on completed Non-AI tasks, I suspect it was less about unwillingness, and more about inability.

We should really be studying AI related cognitive and/or skill decline more heavily.

Don't let the downvotes get you down. They don't know who funds or constitutes METR. If you actually read these papers, you'll find they cite themselves 4-5 times each publication. They also consider "hosting an HTTP server using Python" to be a "long time horizon" task.

The OP cited a METR study... if you don't think they're credible, fine, but it's hard for me to see how it could be intellectually honest to cite them when their research is helpful to your argument and discredit them when it's not. One might even call this approach "fooling yourself".

The OP is not citing their claims, but their commentary. Do you understand why I find that to be different than citing their findings? We're not talking percentages here.

Reply on news.ycombinator.com