Hacker News Reader

Comment by tao_oat | Hacker News Reader

This is my experience too. Many people in my offline, non-techy life seem to get almost upset when I argue that models are good at many things (and getting better fast).

And I kind of get it! If your primary exposure to AI is Google search summaries (from their cheapest models) and shitty AI graphic design, it makes sense to think it's vastly overhyped.

View on HN

Unfortunately it’s not just non technical people, it’s also technical people from my experience. They are usually too caught up in “how it always works” so they can’t think outside of it.

I like how Tyler Cowen puts it: "This is the worst they will ever be".

And some people are still stuck on what AI was capable of when GPT-3.5 was SOTA. They don't realize how much they've improved in ~4 years. Even if the rate of improvement slows down, AI will be tremendously more capable in another ~4 years.

Not my experience, an I'm a heavy daily user of frontier models for software development. Have they improved at all in the past ~2 years? Not really, at least not noticeably. We've improved some of the tooling around it, developed some mitigations for the many systemic weaknesses, and wrapped some things in a for loop to iterate faster. The worst they'll ever be? Maybe, but maybe also almost as good as they'll ever be, looking at the sharply diminishing returns. Reactions to every new model seem to always be roughly evenly split between it being better or worse.

And what do we have to show for it? We can localise some bug causes faster, and it takes some of the mindless boilerplate work off our hands. Which affects maybe 10-20% of our actual job, and even there occasionally eliminates the gains again right away because you have to babysit or manually redo it, without always knowing when and where that might happen. That's not what I'd call a revolutionary increase in efficiency.

And this isn't just my impression, it's what I hear from pretty much all of my professional peers, which includes people waaay more experienced and smarter than me.

This is a surprising opinion to me! Subjectively, I very strongly feel that models improved significantly over the last two years, particularly around the time of Opus 4.6 and now Sol 5.6 and such. It's made a huge difference to how much autonomy I can give them.

If you trust benchmarks (which is debateable) then the rate of improvement is speeding up -- definitely not diminishing returns.

Reply on news.ycombinator.com