It could be that the engineers who are extremely productive with LLMs are landing PRs that look indistinguishable from good, hand-written PRs.
It seems likely that the gains from generating tons of code are being offset by the debt incurred to understanding what you're doing. We see lots of greenfield projects one-shotted with GPT or GLM or whatnot, but very little on the side of projects with long-term maintenance goals. This is telling, to me, that the _effective_ gains are much lower than perceived (it's lots of fun to see the thing crank out code at breakneck pace, probably contributing to this). Still quite nice, and very useful, but not a totally new paradigm.
My guess is that open source folks overindex on being anti-AI.
Eg if you check out Godot, initially they were cool with LLM code, and then had to clamp down on it because it was not leading to anything good
For instance, we had one person submit a merge request to convert the entire GIMP codebase into Python, and it was just tons of files with stubcode implementations. Another person submitted a fix for our OpenEXR plug-in to load YCbCr images, and when I reviewed it, they'd "imported" a non-existent version of the library that had an imaginary "OpenYCbCrOpenEXR ()" function. :)
Wherever these “10000x” engineering gains are… I’m wondering why they’re only for shit nobody sees or touches.
Where are all the killer new things? All we see are glorified hobby projects (guilty as charged) and an unimpressive pace of equally unimpressive feature additions to all the same old stuff.
Businesses with resources out there should tackling huge things we all want, that they never could have undertaken. But I’m not seeing it. Everything is just… more or less the same. And this tooling didn’t arrive last Wednesday.
Or did we get magical AI at a time when we’re also completely out of ideas? That seems unlikely.
This shit just isn’t adding up. We’re not seeing anything that isn’t more “here’s some more ai tools and services… they’re basically the same as the ones you saw last month.”
What for? Someone show me.
I agree they are over-hyped. Maybe the problem is that all the things LLMs are great for are essentially quality of life improvements. There are a dozen killer apps worth of functionality I get out of LLMs - near instant proofreader, one-shot code bugfixer, excellent cursory research tool, bibtext generator, etc. All of these things extend what I can do a bit but none of them instantly result in me being able to generate category killing apps. They still make stupid mistakes, they don't reduce my workload, but they reduce cognitive load on mundane tasks.
Instead, software is plodding along exactly the same as it did prior to LLM code generation, and there's no evidence of superprogrammers making superprojects in 1/10th of the time. With a 10x productivity gain, what used to take a year should take a month
But that doesn’t mean they stick.
And the core issue isn’t AI, it’s that… a successful, highly adopted project requires time for people to know it exists, adopt it, and also time by the maintainer growing community and ensuring reliability. The people factor is more important. I see tons of new projects, but I’m not going to pick a brand new project that has high odds of being abandoned. I’m going to pick the one that consistently maintains it and has some adoption already.
I have seen engineers create very successful internal projects fairly quickly. And yes, a project that would have taken a year taking a month. And offering a lot of extra bells and whistles that you just wouldn’t have time for.
But these are greenfield internal projects, and the bar is much, much lower for those.
I have seen multiple internal incidents root-caused by an agent faster than the humans responding. Just because it can go up and down rabbit holes a lot faster.
The problem is that AI solves one bottleneck, but not others. One team member produces a huge amount of new PRs. (Like 12 solid enhancements and big fixes in a couple hours.) Now I have less time because I’m reviewing that. And we’re all context switching a lot more.
On top of that, I find AI workflows continue to be deeply immature, even though certain models are very effective and very good at troubleshooting.
The story around testing is not really improving for example. AI can write tests, but are they good? I don’t think we have much actually ensuring product quality and reliability automatically. Unit tests are not enough.
Collaboration is very poor too. Coworkers agent creates PR, and now I’m reviewing it, and now he sends my comments back to his agents… really clunky workflow especially since I’m ACTUALLY just prompting his agent.
Plus, certain models (Opus) are getting much worse at writing. I will not use Opus any more because the writing style is so horrible.
The constant change means approaches that worked well a couple months ago don’t work well today. And there are no real experts, because no one’s been doing this for long. And half the posts and learning out there are outdated, or straight up blogspam.
This makes it hard for people to learn and get better, despite the fact that models like Sol 5.6 are effective troubleshooters, and write decent code.
LLMs are superhuman at short term coding such as debugging and writing tests though and you're missing out by not using them there.
Commercial and industrial electricians are LLMs now?