I would expect the models to be bad at, say, division of long numbers in the same way humans are bad at doing the same calculations in their head!
I'd be more interested in having the thing "study maths" in the sense of seeing many examples of Wolfram Language being used in matching English context. That way, it would learn the English->Maths mapping in a format that it can then feed into an industrial-strength mathematics engine. Apparently, Stephen Wolfram is working on this now, but the "full" fine tuning training would require significant funding and time. I suspect OpenAI has other priorities right now, but this type of thing will eventually become a routine way of making specialised and/or more capable large language models.