This already exists: Google's recently-published LaMDA dialogue model [1] is trained to consult a calculator for arithmetic questions and consistently succeeds at it.

[1] https://arxiv.org/abs/2201.08239v2