I'm no expert, but it feels like this exact thing is how it ends up eating itself. For two reasons.
- Eventually the universe of "provably done before" is now mostly LLM genned code. So the training data quality starts a slow decline.
- That it's "good enough" now for 90+% of the work kills the pipeline for any new human software experts. So they never gain the experience and insight to be able to steer the AI to produce ok-ish code. Existing experts age out. This cycles with the bullet above over time. And at some point we're stalled, all code is now mediocre at best and while output is high, innovation/evolution is low. Or whatever innovation we have is coming from a very small circle without much natural competition.