Because it's just cope. Look at the current reality. Are companies rushing to fix bad or even buggy code written by human devs? No, not in most cases. In most cases, if a piece of code "works", it is left the hell alone. And that's the thing about AI code: it does work. The quality is irrelevant in the overwhelming majority of cases (especially if it's other AIs that are adding to it, which is the case more and more often).
Software quality is especially important in safety critical applications.
We should not expect an LLM trained solely on formally-verified code to produce formally-verified code. I don't think that also training on specs and hateful training material will fix that.
So then we're back to the original software engineering objectives of writing better SAST, DAST, Formal Method, side channel, and fuzzing tools for software quality assurance.
Like ~100k people in Meta - nothing critical there, right? Many thousands could be replaced with AI-coders there.
Compare traditional woodworking with modern carpentry on quality, longevity, and marginal efficiency.
From "Why Don't People Use Formal Methods?" https://news.ycombinator.com/item?id=18965964 :
> Which universities teach formal methods?
> Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs?
> Is there a certification for formal methods? Something like for Engineer status in other industries?