This is in a way the same argument that all companies will use just boxed (albeit customized) software. In reality many, ie banks run their own custom software and for good reasons. Wake me up (in retirement) when any major bank decides to run their finances, regulatory stuff and transactions on llm-generated stuff made by underpaid devs like you describe.
We're doing a great job of that right now, aren't we?
LLMs are non-deterministic so any interaction with them is a bug inducing act by definition because you don't know what you're gonna get.
We get about 10% YoY now, which is actually impressive, but not even close to the past experiences. Computing power used to take 18 months to double, and was being attacked at a bunch of different angles.
Now we are literally out of tricks, to the point where we had to undo some of those tricks because they were fundamentally flawed, and it takes 100x the engineering investment to get that 10% YoY, or sometimes only every other year, and that isn't going to continue for much longer.
We can't increase clock speed much. We can't increase density much. We are out of clever tricks. It takes an entire earth's worth of human talent just to produce the machine that produces the chips. How the hell are people still insisting that computing power continues to follow Moore's Law?
Several of the limits we've hit are fundamental matters of physics.
Meanwhile, LLMs ask for orders of magnitude more compute than is cheap and cheerful right now. Do we even have a single order of magnitude left for compute?