The obvious use case here is deep mathematical research, where the LLM can focus its reasoning on higher level concepts.

For example, if it can reduce parts of the problem to some choices of polinomials, its useful to just "know" instantly which choice has real solutions, instead of polluting its context window with python syntax, Google results etc.

Well sure, in theory any mathematical problem can be solved with any Turing complete programming language. I think the idea here is that for certain problem domains Mathematica might be more efficient or easier for humans to understand than Python.