That's not at all how I would assess whether someone understood how to do multiplication – I would ask them to explain their process for performing multiplication or at least to show their work. Maybe I'm misunderstanding what you're saying, though?
Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. (here, the abstraction to people breaks down, because people can walk you through their reasoning - whereas for an LLM, explaining how multiplication works and performing multiplication are very different tasks)
There are weaknesses to this approach: you get a binary yes/no, and you have no idea how close they got. You can't tell if they just made a small math error or if they don't even know what numbers are. Going back to the LLM setting, this is why a continuous metric is useful, compared to one that experiences step-function behavior.
Right, but unless you know something about their process, the only way you can determine whether they have the ability to (correctly) multiply two arbitrary four-digit numbers is to have them demonstrate multiplying every combination of four-digit numbers. One can easily imagine a system that gets most answers correct but fails certain cases (e.g. only carrying between pairs of digits).
The reason why an inexhaustive test (using only several dozen examples) works to some degree with schoolchildren is because we know something about their method: the algorithm for multiplication they've been exposed to – that they're explicitly being taught – is an algorithm that we know to be correct.
That has a very large effect on the likelihood of different failure modes and the evidence required to have a particular degree of confidence.
Furthermore, for "system 2" activities (to borrow the term from Kahneman), we can reasonably expect a person's description of their process to match the process they actually performed. (There are exceptions: people will produced incorrect post hoc explanations for their behavior under sufficient duress when "I don't know" isn't perceived as an acceptable answer.) But I'm personally not aware of any reason to believe this about LLMs. I don't know why the network's actual process for performing multiplication should have anything to do with the text it produces after the fact when asked to explain its work.
So I'm not debating that a reasonable way to guess whether a person can (correctly) multiply two numbers is to "ask them to multiply two numbers and see if they get it right"; I'm disagreeing that this works with LLMs.
Did Khaneman get into this in his book?
Also, as for the point of contention, I think if you tell the LLM to show its work in mathematically formal notation, it's far more likely to be able to produce correct answers (I think there was a post on here in the last week or so demonstrating that?). I think this kind of makes the comparison to humans more fair, because inside their mind humans are doing some sort of intermediate math in their head for anything beyond trivial problems, and an LLM needs to be able to speak explicitly to compete fairly with that (I speculate/propose).
Not from what I recall; this was just my own observation. (Also, in retrospect I should have made it clearer that this exception was meant to be an example more than a particular claim... "under sufficient duress" almost makes it vacuously true.)
> Also, as for the point of contention, [...]
Yes! I've heard this reported as well, multiple times. I suppose it's sort of like using a pen and some scratch paper.
The model doesn't "intend" to multiply two numbers together: it has been equipped to parrot what a person asked to multiply two numbers together might say. If we asked it how it performs multiplication, it is going to produce a process it thinks a human would claim to use to perform multiplication, but that doesn't mean it can actually apply that process.
The claimed "emergent" property is that a model that can successfully fool humans into thinking they are talking to a person quasi-magically involves becoming "good" (for some measure of "good") at the cognitive tasks humans are capable of. This paper suggests that the measures of "goodness" researchers have been using make the gains on those cognitive tasks look more dramatic than they would be if measured via linear metrics.
I suspect some of the disconnect expressed in these comments here is based on the participatory nature of being lied to by these models. The reader is a full participant in creating meaning from the output of a model. Even when the improvement in models is linear, our willingness to suspend our disbelief is not. Especially when we want to be fooled: it has to avoid anything that would jar us out of our belief rather than proactively and repeatably succeed at cognitive tasks, and that is a non-linear measure.
Citation needed.