Hacker News Reader

Comment by qsort | Hacker News Reader

> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM

I think there are two different claims here:

- developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims.

- LLMs give no productivity uplift at all, which is much harder to defend. A repeat of the famous METR RCT study did find evidence of improved productivity, and this seems to align with the experience of many experts I trust.

View on HN

Specifically my claim is "the relatively minor productivity uplift I would personally get out of agentic development is offset by the high cost, along with unresolved questions about long-term code maintainability, so I am not convinced that it is actually beneficial."

IMO the bigger problem is that ~1.5x individual dev productivity uplift seems to translate into 1.05x uplift across the team. People have been waaaaayyyyy too overconfident about this stuff.

I am both a career developer and experienced team manager. from first hand experience the 1.5x im getting from AI is not flowing down to my team / org because why would i output 50% more when the pay environment and leadership are already underwhelming. That additional 50% productivity goes completely to side projects built on my second computer between 9-5 tasks

Actually, the METR report speculates that some of the overreported productivity uplift comes from grabbing unnecessary low-hanging fruit, things like "oh I'll make a web dashboard to keep track of this stuff // wow that would have taken all day without Claude!" But in the olden days they would have just used a notepad. Yet psychologically they built a real thing and saved a lot of time.

True I have made a few low tier apps that just hit apis that were previously obfuscated deep in menus that have had an outsized impact

> But in the olden days they would have just used a notepad. Yet psychologically they built a real thing and saved a lot of time.

There was an old Onion article about the monetary savings from buying clothes on sale as computed by (a) the women buying the clothes; and (b) their male relatives.

Your comment makes total sense to me but generally, in regards to productivity gains from AI, I can never understand where these are realized for people. Maybe I'm just a laggard but never found myself 25%-50% behind on anything or that much more work/items/tickets available.

That’s what I’ve heard from my dev team too. They’re using it to give themselves free time while still being on the clock, not to produce more output for the company. Roughly thinking about hours spent on projects I think have gone up per task, the opposite that should be happening.

I'm finding it almost impossible to fill that free time with work, unless it's just reading emails and chat messages.

I can context switch between two or three chats, but doing so speeds up my agent use at the cost of making reviews and discovery harder, so it might come out in the wash.

>I can context switch between two or three chats

I'm finding this is a skill which I'm slowwwwwwly improving but, for now, my level means I tend to miss mistakes I'd notice if just working on one task. I also had to make accommodations to the way I work to make it achievable, including telling my employer that a 500 GiB disk just isn't enough any more when I need multiple builds in parallel.

Great point. Devs have relatively no incentive to be more productive for their orgs, and all the incentive to be more productive on their own personal work. The AI benefit to devs is real, just not for large enterprises IMO outside of automating clerical/mundane work.

> Devs have relatively no incentive to be more productive for their orgs, and all the incentive to be more productive on their own personal work.

AI definitely supercharges the side project myth, but I think we’re going to see waves of developers quitting or losing their jobs because they’re chasing dreams of running a side code project that isn’t going to turn into a business.

The incentive to be more productive at work is that they get to keep their job and not be replaced by a cheaper junior. I think we’re in for a reckoning across the industry as companies realize that there’s little difference between a lazy senior armed with Claude and a halfway motivated junior who has aspirations of growing into something more. The latter costs less and might grow into a better dev.

If the salaries fall then there would be no highly motivated juniors.

Lazy senior vs motivated junior tension has always existed. I was one then became the other. I don’t see llm changing this dynamic!

> because why would i output 50% more when the pay environment and leadership are already underwhelming

The productivity uplift measured in some of these reports is from more tasks being done, not from developers giving themselves more idle time during the day.

If we do see job losses from AI, my feeling is that it’s going to be concentrated in people with an attitude like this one where workers try to remain anchored to their old delivery pace. It’s becoming easier than ever to replace a worker in that position with a lower paid junior equipped with Claude who has an interest in learning. There’s no reason to keep a higher paid worker around to copy and paste between Jira and Claude.

thats absolutely true regarding wysiwyg projects like a small website. aside from temporary cost savings, systematically ignoring institutional knowledge in favor of mass output is not gonna do humanity any favors ..

Yeah institutional knowledge is hard to capture, even more with agentic coding.

I just compressed what would take a team a few months to deliver in one or two months, the amount of gotcha I spot from the existing system we work on everyday is staggering.

How to deliver and keep these feedback is difficult, the other teams are overwhelmed by other tasks, so any tickets or communications will be forgotten in a few months.

And as an external ressources allowed to use frontier model, I have a hard time to keep up with my own pace, and all that institutional knowledge will be lost both by my employer and client.

Could it also be that the phrase “using an LLM” is loaded in itself? Who is getting more velocity out of it:

- chat box interaction

- vendor harness user (the cursor / antigravity crowd)

- SDD Claude users

- bespoke harness creators who let it goooooooo

I mean we already know who’s using all the tokens and driving enough cost to make penny pinchers consider removing other “costs”

I have seen no productivity uplift at all from LLMs. They are at least about neutral these days (they used to be a productivity drain), but no gains at all. By the time I get done reviewing the code to make sure it hasn't done anything crazy, I've spent the same amount of time I would've taken to write the code myself. The only people I personally know who claim productivity gains are getting those gains by completely disregarding quality, or understanding the code, and just YOLO letting the LLM do everything without checking. I'm not willing to do that.

I find Claude is shockingly slow. If it were genuinely instantaneous, it would be worth it, but I often find myself twiddling my thumbs while waiting for it to cook.

Yeah that's why you do multiple tasks and projects simultaneously.

Humans notoriously suck at context-switching. I doubt you can really do multiple projects simultaneously without compromising quality.

You're already vibe coding, how worse can it get?

That sounds like a really efficient way to very quickly produce a mountain of crap.

> LLMs give no productivity uplift at all, which is much harder to defend

It’s not really hard to defend. Because when people says that productivity is uplifted, they are talking about amount of work, not the ROI. That’s why you keep hearing about LOC, amount of PR and prototypes, and the time taken is actually “time to PR” and not “time to production + time spent on bugs”.

There's another even more abstract liability on the balance sheet here for "deprecated LOC" or something, where the thing they built gets scrapped because no one wants to untangle the rat nest, or the "documentation" was so incoherent that it all had to be rewritten and reviewed.

Coming from legacy enterprise I am using claude to do exactly this for some of my team's more bizarre and poorly documented "inventions" that started out as their pet project and then became part of our official workflow.

Maybe we can make an analogy with taking the plane vs driving to your destination.

A plane literally goes 10x faster than a car, so a 5 hour drive becomes a 30 minutes flight. But if you have to drive to the airport, arrive early, pass check-in, security, boarding, then pick up your luggage on arrival, rent a car and drive to your destination, you may realize that your 30 minute flight took you more than 5 hours in total.

You also have to consider the Amdahl's law: a 10x speedup on 10% of the project is just a 9% speedup overall.

> You also have to consider the Amdahl's law: a 10x speedup on 10% of the project is just a 9% speedup overall.

This is totally true for me. I'm working on more features/bugs at work and getting the coding done faster with claude but the coding part is at maximum half of the process and usually less. The rest of it is clarifying the requirements from our product team or given that we're a very legacy place, what the existing state of our infrastructure and other systems are for the new changes to fit in to, etc.

> You also have to consider the Amdahl's law: a 10x speedup on 10% of the project is just a 9% speedup overall.

It's a 10% speedup, unless you think that a 100% speedup would reduce completion time to zero rather than cutting it in half.

Assuming that "10% of the project" is measured by time requirements, when you get a 10x speedup on that part of the project, you'll end up completing 100 units of work in 91 (formerly 100) units of time. Your work rate is then 100 / 91 = 1.099 times what it was before; that's an improvement of 10%, not 9%.

(Sanity check: suppose you have a project that will take 100 days. That project is reassigned to someone 10% faster than you. They will take 100 / 1.1 = 90.91 days to finish.

If they were 9% faster, they'd take 100 / 1.09 = 91.74 days to finish.

Does a savings of 9 out of 100 days of work look more like the project that saves 9.09 days, or the one that saves 8.26 days?)

In the former case, which seems more likely, does it not fall to reason that like any tool there'll be some that will benefit from it and some that it just won't work as well for, but are equally as good at their job? Like that neovim, unless your an absolute zealot you wouldnt insist that everyone use it, or hire someone based off it, but those who do work well with it do find it to be a boost to productivity.

Also importantly, can it be a neovim? Neovim hasn't had literal trillions invested over it in the space of just 5 years, local llm's aside can those smaller productivity gains justify the huge investment put into them, and which will continue to be needed for further model development?

Developers are cashing in on the productivity gains. Meaning instead of using the increased productivity to do more work, they just become lazier or do fun irrelevant side projects instead, where as before there was just not much time for such things. I have definitely procrastinated on work simply because I know I can swoop in with an LLM and do it all in 15 minutes, whereas before I would have spent a few hours.

This is how LLMs can both result in greater productivity yet still not appear to yield much more benefit than the pre-LLM era.

And there is no benefit to the developer for doing the work first then just sitting idle. That’s how you get people putting more things on your plate.

I would frame it differently ... Last year I was burning the fuck out hard. This year I am not.

My gain is that I produce good work in a normal workday and I am absolutely not writing code to 4 am anymore...

Overall for me it is a win for me and my company.

Odd, it was the opposite for me. Last year, no burnout. But then agents got good and everyone wanted more code more features faster and faster. I got burnt to a crisp. Now? I don’t give a fuck. Ok people want to push tons of code straight to prod? Go ahead, looks good to me… no need to review it if it’s with the latest frontier models right?

Some unsolicited, probably obvious advice for you for the future: never do that again.

Managers will abuse you if you expand your workday to make sure deadlines don't slip. They will see that the work is getting done and see no urgency to hire more help.

You have to surface capacity issues to management, learn to say "no" diplomatically, and/or "I have capacity for 2 things out of these 5, prioritize which ones you want first."

The LLMs are better at enforcing their boundaries than developers. Run out of tokens? That's it. No more code. It doesn't care how much the company needs this for the customer call tomorrow.

I feel that’s the opposite of the LLM-business models.

If I run out of tokens I’m encouraged to spend money to buy more tokens, Claude has an overage setup, and Anthropic runs occasional token sales.

So, to that example: the more often customers are hitting token limits under duress the more likely to open up the wallet and pay to finish.

So, its drug dealer economics.

Run out of your fix? (Tap tap) Buy some more!

Exactly, I finally migrated that old service to typescript and cleaned up some really annoying tooling, for example. I’m more productive and can get quick fixes out very quickly. We’re also seeing agents help identify incidents faster than human investigation can. But big structural things that are hard to integrate still take lots of time.

A few months ago, I heard a keynote speaker claim that LLMs did not give any measurable productivity gains, except in two fields: customer support and software development, where they resulted in about 30% productivity improvement. I forgot if he presented any sources, and if he did, I forgot those, but it sounds plausible to me.

But instead of productivity, I'm much more interested in using it to improve quality. You've got a tireless reviewer who is always ready to review your code and catch any gaps.

I don't know how they're measuring productivity for customer support, but it certainly isn't in terms of customer satisfaction. People have never been happy with chatbots when they're trying to reach a human.

We are getting ai responses from cloud providers. From both aws and google, my teams have received obviously ai generated and non-applicable advice.

The latest was that we need to write in object pooling in their client code, an option that doesn't exist yet in _their_ client code. Like, yo, that is a you-problem on performance bottlenecks; we expect you to provide solutions.

Reply on news.ycombinator.com