A precise and formally verified specification is WAY harder to write then the code for it
The software industry has gotten this far with very little help from the formal methods community... that's changing in recent years in certain spaces where errors are magnified by scale like cloud computing, etc.
But instead of getting better at writing precise specifications we're going to continue to be bad at it and hope that an LLM can manage to infer the correct program. It might be millions of lines of code but hopefully it does some of the things we want most of the time.
Update: To be clear, I'm not saying AI/ML programs cannot help us to write programs at all, just that the inputs need to be better if we're going to have any confidence that the programs it generates are any better than a horoscope.