It's far easier even for an expert to communicate what they want in natural language than it is in a formal syntax for all but the most trivial things.
It should be a goal of these tools to do this correctly.
agree completely, but the LLM should be focused, then, strictly on formulating an "execution plan" of sorts and handing that off, not on performing math itself.
In other words, when asked "if i have 349 blueberries and one blueberry turns into a cherry per hour, how many of each fruit will I have in 93478 minutes?" it shouldn't be doing the actual arithmetic, but it should be figuring out what arithmetic would need to be done.
You describe a complex relationship between several nodes(people, cities, etc), and then ask ChatGPT to draw the relationship as a graph data structure. It will create a formula that Mathematica can render, and then send the formula to Mathematica before presenting an ascii drawing. Usually. Sometimes it just complains that it can't draw and explains what the graph looks like to you in a written formal/human-readable syntax.
In other words, yes it just summarizes the problem, converts it to a formal syntax, and sends it off to some other tool.