Feed forward Neural Networks, like LLMs using attention are effectively Directed acyclic graphs, so it has little it can do to 'evaluate from the outside'
Even if you can move to second-order logic logically-valid formulas in second-order logic is not recursively enumerable. AS RE problems are those which a Turing machine can answer "YES" in a finite amount of time, but a "NO" answer might never come, moving out of RE spaces like DAGs is not computationally feasible.
The problem with 'pattern finding' is that it can learn some even HoL problems if it is in the corpus, but will fail on even simpler problems that weren't.
Here is one paper that explains this. https://arxiv.org/abs/2301.09723
"A persistent problem in corpus-based ML, in all its applications, is that the patterns that the AI finds do not actually reflect the fundamental characteristic of the problem, but rather superficial regularities in the training data, known as “artifacts”.
LLMs aren't doing the 'logic' of the math problems, they are finding patterns in their training data that are hopefully close enough to work for the presented problem.
This is why you can use AI to say learn about intervals on the real line, as those are of finite VC dimensionality, but algebra questions that are outside of it's corpus tend to be very difficult for LLMs to be correct on.
And obviously issues like the Entscheidungsproblem don't magically go away because we have a tool like ML that is far more computationally efficient than brute force, but still insufficient.
As LLM's will confidently present wrong answers as correct, how is that helpful for students?