e.g. “x is a variable. y is equal to two plus x. if x is two, what is y?”
and when run, it outputs four. what would be the implications of this?
Alternative answer: "y is a variable".
To the essence of what I think the original question is asking: It seems possible (due to the turing completeness of some NN architectures) that a language model could learn a sub-network that was a turing complete computational framework + problem solver and then use that to solve problems in natural language. But it also seems highly unlikely that it would actually play out that way.
The linked paper has a nice summary of attempts to build neural nets that learn algorithms. It seems this more direct approach would be likely to yield a result faster than waiting for such a capability to emerge in GPT-12 or something.
Of course most ML models execute from within a turing complete programming language/execution environment anyway. If we want to solve math problems, it might be easier to train a model to translate natural language into a more formal definition (afaik this is still very hard) and then hand off the formal definition of the problem to an automated theorem prover. After all, being turing complete doesn't make all problems solvable or tractable from a run time / memory perspective and that's probably the bigger bottleneck when it comes to solving hard math problems.