> haven’t seen anything suggesting it’s doing special-case experts for maths
I didn't say it is. I said it is at least trained on images, which means it has a visual processing layer. I then mentioned that in humans, the visual sensory memory used for persistence-of-vision — along with the higher-level abstract positional memory used for navigation and not tripping on tree roots — has been shown to be active when doing arithmetic; and that this is suggestive of the visual field being used to "outsource" positional/sequencing tracking for numbers.
My implicit hypothesis (that I didn't want to say explicitly, because I'm not an ML researcher and I have no idea how to even begin to determine the truth-value of this) is that the GPT architecture is able to be as numerate as it is, vs. other pure text-in-text-out language models, because it's reusing the generalized visual field it evolved to map images into tokens, as a within-inference-step working memory for holding absolute token positioning meta-information. (Or, to put that in human terms: it's visualizing the numbers.)
> But it’s not required for basic numeracy, “tricks” as you put it, or whatever else GPT has learned, can objectively get us to median human performance.
No — as the median human (with a pencil and paper) can do simple arithmetic on arbitrarily large numbers.
The difference between "memorizing a bunch of tables" and numeracy is that numeracy is a knowledge of algorithms, not a memorization of truth tables; it a set of skills that can be applied to never-before-seen mathematical objects to yield correct answers. You can ask a human to compare two 800-digit numbers, or add them together, and they'll be able to do it, one step at a time.
As far as I know, GPT does not have the "skill" of numeracy in the sense of being able to do even simple arithmetic on unbounded-length numbers. And I don't mean the boring thing (that it has a bounded context window, so the number has to fit in there); I mean that it fails at adding two numbers when you start to get up to even just e.g. 64-digit numbers. It starts doing things like (seemingly) breaking the numbers down into sub-sequences and independently adding them up, but then forgetting to carry between the sub-sequences, or even forgetting which order the aggregates of the sub-sequences should be put back together in.
It seems very apparent to me, after much experimentation, that GPT models are just trying to treat numbers as a finite set of objects (maybe 100K-or-so?), each with a set of baked-in properties and relationships — plus a set of logically unsound rules they've derived for breaking large numbers down into small numbers, and putting small numbers back together into large numbers. These models are, in other words, using language skills (memorization of properties; adjective grouping; analogy) to pretend to do math — to cargo cult a symbolic-manipulation process they don't understand, in the hopes of at least looking like they're doing it correctly — but that's not the same as actually applying the scalable process of arithmetic to an arbitrary number.
An adult who "did math" this way, would be described as "someone who never learned to do math." And they would, indeed, be considered innumerate. (Could they do their taxes? Split a bill? Make change? Determine which of two products where one is per-lb and the other is per-each has the better value? No? Then they can't get by in society. That's innumeracy!)
---
But also — to pop the context here: we're not talking about GPT. We're talking about a different language model (Llama 2), that's very likely strictly worse than any of the GPT models are at math (though I'd be intrigued to be proven wrong.) I assert this because, as I said above, I believe that GPT is as numerate as it is because of its visual sensory field — which the Llama models don't have. Thus my initial assertion: if even a multi-modal language model like GPT isn't close to full numeracy, then a pure language model has no chance at even vaguely simulating numeracy. And that that's why the OP is seeing the errors they're seeing.