LLMs can in fact do arithmetic, just not reliably owing to how numbers are represented probabilistically: https://arxiv.org/abs/2410.21272
I just asked ChatGPT to multiply two 4-digit numbers, and two 7-digit numbers without external help. It got both right. I'm sure it wouldn't have a 100% success rate, but saying it can't do arithmetic is just false.
Are you sure it honoured your stipulation of "without external help"? For all we know, it hacked its way into Wolfram Alpha and got the result from there.