I don't think LLMs are going to be able to solve that. There are a number of things that are assumed are true, but may not necessarily be true. This can potentially lead to multiple possible answers (outputs) given the same inputs.

For example determinism in code, its required for computation and its a system's property, but generalizing a test for it is really hard. Its a property, and by knowing its true or false you can make inferences on whether a system maintains those properties, but most of this is abstracted away at lower levels and since the context can't ever be fully shared with an LLM for evaluation, nor can it automatically switch contexts when evaluation fails, this most likely will never be solveable by computers when there exists one single input that produces two separate (different) outputs, at least from what I know about automata theory and computability.

Its generally considered a class of problems that can't be solved by turing machines.

https://en.wikipedia.org/wiki/Theory_of_computation

https://medium.com/@tarcisioma/limits-of-computation-231bf28... (overview)

https://en.wikipedia.org/wiki/Undecidable_problem (crux of the problem)