I don’t disagree that these are useful tools, by the way. I just haven’t seen any discernible uptick in general software quality and utility either, nor any economic uptick that should presumably follow from being able to develop software more efficiently.
There are many really lucrative markets that need a fresh approach, and AI doesn't seem to have caused a huge explosion of new software created by upstarts.
Or am I missing something? Where are the consumer facing software apps developed primarily with AI by smaller companies? I'm excluding big companies because in their case it's impossible to prove the productivity, the could be throwing more bodies at the problem and we'd never know.
The challenge in competing with these products is not code. The challenge competing in lucrative markets that need a fresh approach is also generally not code. So I’m not sure that is a good metric to evaluate LLMs for code generation.
I could see it being doable by forking LibreOffice or Calligra Suite as a starting point, although even with AI assistance I'd imagine that it might take anyone not intimately familiar with both LibreOffice (or Calligra) and MS Office longer than a weekend to determine the full scope of the delta between them, much less implement that delta.
But you'd still need someone with sufficient skill (not a novice), maybe several hundred or thousand dollars to burn, and nothing better to do for some amount of time that's probably longer than a weekend. And then that person would need some sort of motivation or incentive to go through with the project. It's plausible, but not a given that this will happen just because useful agentic coding tools exist.
So where are they?
We're not asking to evaluate LLM's for code. We're asking to evaluate them as product generators or improvers.
I can go to Wendys or I can make my own version of Wendys at home pretty easily with just a bit more time expended.
The cliff is still too high for software. I could go and write office from scratch or customize the shivers FOSS software out there but its not worth the time effort.
So, where is that in the 2020s?
Yes, code is a detail (ideas too). It's a platform. It positions itself as the new thing. Does that platform allow upstarts? Or does it consolidate power?
We have superhuman coding (https://news.ycombinator.com/item?id=45977992), where are the superhuman coded major apps from small companies that would benefit most from these superhumans?
Heck, we have superhuman requirements gathering, superhuman marketing, superhuman almost all white collar work, so it should be even faster!