> I'll risk sounding antagonistic and ask you this: if LLMs are not "good enough", why are they still around?
To quote the parent commenter:
> They seem to be able to solve the immediate problem,
but not long term
To quote the parent commenter:
> They seem to be able to solve the immediate problem,
but not long term
No quality large scale project can exist without it. And any seasoned engineer should understand by now that no SOTA LLM can produce quality engineering at scale.
Execute fast or get nowhere, a series of small wins get you to live long term.
If you need examples, look at game dev. There's plenty of games that have good execution but aren't fun to play or they're a confusing mess because there wasn't good overall direction.
Just because a videogame isn’t fun doesn’t mean there wasnt a cohesive engineering vision. God of War: Ragnarok is a good example of this. Excellent technical execution and (arguably) great direction but a terribly boring game.
And many games that are terribly engineered are also incredibly fun. Dark Souls is a good example - runs like ass, looks very rough even by the standards of the time when the game released, with terrible enemy AI in a combat focused game - one might even call it a confusing mess (and I laugh at the notion that it doesn’t have good direction)… despite all this it’s a classic and arguably the most influential game since its release.
Elden Ring carries a lot of legacy baggage from years of iteration on the same engine as ~~Dark~~ Demons Souls - and it’s widely considered From Software’s magnum opus and a masterpiece. Yet considering the scope of the game it is a great example of how small wins added up can solve large problems when executed with the skill and judgment gained from experience. So one might still consider it a success in terms of “engineering”.
Would From Software have been able to make the same game using Unreal Engine 5 built from scratch with dime a dozen “Unreal experts” brought in off the street? Not a chance. Want to know why western game development studios are suffering right now? Compare American layoff culture and race to the bottom economics to the retention rates of Japanese game studios like Capcom, Nintendo and other heavy hitters.
For many of the most marvellous things humans have ever built, if you take a peak behind the curtain you’ll still probably find some amount of duct tape and popsicles sticks somewhere keeping it all together.
Fun, however, is a matter of taste and you can’t engineer that.
… but does it work? Someone comes along, reviews it, "this is garbage, and does not do what it says it purports to do". Perhaps it even gave an output: the script computed … something, but it's just GIGO.
But that "check if this works" friction is the same friction that is what people try to avoid by generating it with an LLM in the first place. If you're too lazy to write the script, you're practically by definition too lazy to verify it.
Sooner or later you run out of turtles to put on the stack.
I literally can't imagine generating a script with an LLM without testing it at all.
Bash is one of those situations where LLMs can do really well. No human on Earth can remember all of the commands and even fewer humans can remember all of the switches for all of the commands.
This kind of remembering, searching, and assembling is exactly what LLMs are good at - as long as you're not writing a gigantic build system with hundreds of moving parts, in which case you should probably be using something more streamlined anyway.
Then you're extremely unimaginative as well as unusually fastidious.
Certainly someone - several someones - are generating lots of scripts and not testing them, given the PRs I'm seeing.