They can. They're sometimes a bit cocky about their maths abilities, but this really isn't hard to test or show.
https://gist.github.com/IanCal/2a92debee11a5d72d62119d72b965...
They can also create tools that can be useful.
Of course the model will use the calculator you've explicitly informed it of. The article is meant to be a critique of claims that LLMs are "intelligent," when, despite knowing their math limitations, don't generally answer "You'd be better off punching this into a calculator" when asked a problem
> Of course the model will use the calculator you've explicitly informed it of
I didn't. I also gave it no system prompt pushing it to always use tools or anything.
It searches for tools with a query "calculator math root" and is given a list of things that includes a calculator. It picks the calculator, then it uses it.
The code and trace are right there.
The vast majority of features that Anthropic and OpenAI ship are just clever ways of building system prompts
Even so, doesn't informing the model of the fact that some "tools" are available, immediately before asking it a math problem (that would be virtually impossible for a human to answer precisely), seem like a pretty big hint that it should inquire if a calculator is available?
Here's what I get from sonnet in response to the plain user-prompt "What is the eighth root of 4819387574?"
""" Let me solve this step by step.
To find the 8th root of 4819387574:
1) The 8th root of 4819387574 means finding x where x⁸ = 4819387574
2) This is a large number, but it's a perfect 8th power.
3) One way to approach this is to find factors: 4819387574 = 13⁸
4) Therefore, the 8th root of 4819387574 is 13.
To verify: 13⁸ = 13 × 13 × 13 × 13 × 13 × 13 × 13 × 13 = 4819387574
The answer is 13. """