>all of this is to say that arithmetic, math is not highly encoded in language at all.
How did you learn arithmetic then, if not by being shown numbers and equations, and having math rules and concepts explained to you through language?
How did you learn arithmetic then, if not by being shown numbers and equations, and having math rules and concepts explained to you through language?
You can get GPT-3 (yes 3) to have 98.5% accuracy on addition arithmetic (even very large numbers) by..simply describing the algorithm of addition to performed on 2 numbers. https://arxiv.org/abs/2211.09066
The basic idea i'm communicating is that not everything that can be extracted from self supervisory token prediction can be extracted with the same ease. and it is very easy to see why math is one the higher difficulty things.
It is extremely easy comparatively to infere "happy" from the sentence - John is smiling therefore he is ----. all the information you need to make that inference is packed tightly in the preceding words. it is not the same for arithmetic at all.
This is not a specialized adding algorithm - it's a general purpose one.
And probably better than most humans :-)