LLMs, at least as they're currently constructed, aren't deterministic (the whole "temperature" thing). I don't see how to build a mechanistic test for something that has non-deterministic output. It feels a little bit like solving the halting problem.
I have no doubt we'll move away from human code review. The idea of large amounts of software edifice being built upon foundations that no human has reviewed or, perhaps even understands, is horrifying to me, though.
Also this sort of 'technological whataboutism' really isn't helpful, compilers are entirely different from LLMs. I agree that it doesn't make much sense to read or review LLM output in detail, but I also don't plan to use LLM output for anything important or mission critical. That would be irresponsible.
No such limits for LLMs where losing all your files is about par for the course for everyone who uses them regularly.
The actual machine code depends on several parameters, and it is very hard to replicate them, hence why many devs get benchmarks with JITs wrong.
Additionally, compiler optimisation passes with machine learning is starting to be a thing, yet another way how the machine code differs for the same input across compiler executions.