Hacker News Reader

Comment by lbrito | Hacker News Reader

>undeniable, massive productivity gains.

How can something so undeniable have zero scientific evidence? Are there any large peer reviewed or meta studies confirming your claim?

View on HN

It’s a very hard experiment to run. You have a population that’s already “treated”. You can’t blind them to the fact that they’re using AI tools. It’s hard to imagine a study that wouldn’t have serious flaws that people would then use to dismiss and form their own conclusions. Sure you have METR but that was very low n with a very old model.

I think the surest sign of productivity gains is the sheer volume of adoption. If you look beyond headlines, adoption is just incredible. Of course adoption does not necessarily point to productivity gains, but if this was some sort of FOMO or smoke and mirrors you would not see this much retention and this feverish a pace of adoption. You would not see a large segment of the profession using coding agents exclusively. All of these companies track productivity, again with imperfect proxies, yet everything points to a pretty consistent picture. Same with benchmarks, again a lot of crappy benchmarks but a lot of high quality ones too and a very diverse collection of tasks and capabilities they probe.

Your second paragraph appears to be 3 different instances of saying "X does not necessarily point to productivity gains... but in the case of AI, X definitely means productivity" without really saying why that is true or why other explanations do not fit.

Adoption meaning productivity supposes there are no other dominant factors for the AI push nor AI retention. It is possible for practices to be picked up or continued in spite of causing productivity DROPS. What studies have suggested are factors that make for productive work environments and what is actually enforced in the workplace are different things.

It’s 3 different weak but complimentary proxies. We form beliefs from imperfect evidence and I find these fairly convincing when it’s hard to find any hard evidence of no productivity and exactly the scenario you would expect under the hypothesis that we do see productivity gains. None of this is supposed to be unassailable. I would challenge then if you disagree what the evidence you have for this is?

Adoption implying at least some significant productivity gains doesn’t contradict there being other factors. You’re seeing entire companies reshaped. The argument is this is all for show or CEOs are in some sort of idiot class?

“It is possible for practices to be picked up or continued in spite of causing productivity drops” well of course. I just find that incredibly far away from Occam’s razor.

My point is: we have lots of evidence that’s highly consistent with real productivity gains, and I don’t see many pieces of evidence to the contrary.

Sheer volume of adoption is fairly forced though - "use it or you're fired, and tokenmaxx the hell out of it". Most the people I know outside of tech don't seem to be particularly captured by it, if they use it at all.

Yes since frontier perf is too expensive. Consider though frontier perf costs roughly 40x less per year. So you see super adoption for mega costs by the companies with the scale and budget for it, and then folks who can’t afford it that have to wait until maybe fall until they have the game changing perf we saw for opus 4.5 that spurred the enormous adoption by the major players.

I agree that there are incentives to waste tokens but this is strategic: they spend now for people to build and explore, then they keep the things that work and drop the things that don’t.

Because even in a field like software engineering where the output of our work is save in version control, measuring baseline productivity is hard.

LoC: people argue it’s not what’s important

PRs/day: same as LoC

Getting projects done faster: oh but what about the quality.

Solve the technical problems and actually be more productive, the social systems build around the old way of doing things will hole you back.

Finish a PR in 10 minutes doesn’t matter if you’re waiting days for a human review.

Reply on news.ycombinator.com