The other main issue that I see, is that even if there is a formally verified specification, at the moment, LLMs will not respect it perfectly. As long as LLMs are not able to non-deterministically follow a spec, the technology is not good enough.
A part from that, imo, in this age we should focus more on the mathematical aspect of computations, and I think we need to develop novel theories that take into account the non deterministic nature of LLMs in the process. I'm not sure this will ever work by merely extending current practices, as software design practices are extremely poorly defined from an engineer point of view. Just extending them by including randomeness does not seem a good idea.