We also know very well that human learning and GPT-3 learning are nothing alike. We don't know how humans learn exactly, but it's definitely not by hearing trillions and trillions of words.
GPT-3 is doing just that, and then trying to remember which of those trillions of words go together. This is so obviously entirely different from human reasoning that I don't even understand the contortions some go not to notice this.
The core of the difference to me (admittedly not an AI researcher) is intentionality. Humans conceive of an intention, then a communication. This is not what models like GPT-3 do as there is no intentionality present. GPT-3 can create some truly freaky texts but most that I've seen longer than a few sentences suffer from a fairly pronounced uncanny valley effect due to that lack of intention. It's also why I (again, recognizing my lack of expertise) think expecting GPT-3 to do things like provide medical advice is a fool's errand.
It's like if you were building a physiological model of the human body using only skin and exterior features as training data. We would not expect the model to learn the structure and function of the spleen. By analogy we should not expect GPT-3 to learn the structure and function of thought.
It is irrelevant when folks point out "but human cognition isn't any different at its base than the machine" because we can very clearly see there is a massive qualitative difference in behaviors, and there is a wide gulf in architectures and development that there is no reason whatsoever to expect a qualitatively unique (so far as we can tell) behavior as conscious language use to ever emerge in a computer model. It's pretty remarkable that the only other cluster of biological systems that can even physiologically mimic it are songbirds/parrots who come from a very very different part of the phylogenetic tree. Who could ever predict that aberrant homology if I just gave you the 4 nucleotides?
More is different. You don't get complex structure by just crudely analogizing and reducing everything to base parts.
[1] Anderson, Philip W. "More is different." Science 177, no. 4047 (1972): 393-396.
You have a GPT inside your brain - just start saying words - the first thing that comes into your head. These are the words of your statistical model of the universe, your internal GPT-3. read what you have written back - it will make sense, because it is not just a parrot, it is your subconscious.
If we look at anything more specific than 'mathematical function', our brain is nothing like GPT-3. GPT-3 is not trained on sensory data about the world, it is trained on letters of text which have nothing directly to do with the world. The brain has plenty of structure and knowledge that is not learned (except in a very roundabout way, through evolution)*. GPT-3 is lacking any sort of survivalist motivation, which the brain obviously has. The implementation substrate is obviously not even slightly similar. The brain has numerous specialized functions, most of which have nothing to do with language, while GPT-3 has a single kind of functionality and is entirely concerned with language.
And even if I start writing down random words, what I'm doing is not in any way similar to GPT-3, and my output won't be either. It will probably be quasi-random non-language (words strung together without grammar) vaguely related to various desires of my subconscious. What it will NOT be is definitely not a plausible sounding block of text that I determine to resemble as closely as possible some sequence of tokens that I observed before, which is what GPT-3 outputs.
I do not have an inner GPT-3. The way I use language is by converting some inner thought structure that i have decided to communicate into a langauge I know, via some 1:1 mapping of internal concepts to language structures (words, phrases). In particular, even the basics here are different: letters, the things GPT-3 is trained on, are completely irrelevant to human language use outside of writing. People express themselves in words and phrases, and learn language at that level. Decomposing words into sounds/letters is an artificial, approximate model that we have chosen to use for various reasons, but it is not intrinsic to language, and it doesn't come naturally to any language users (you have to learn the canonical spelling for any word, and even the canonical pronunciation; and there is significant semantic info not captured directly in the letters/sounds of one word, through inflection, or tonality and accent; or in sign languages, there is often no decomposition of words equivalent to letters).
* if you don't believe that the brain comes with much knowledge built-in, you'll have to explain how most mammals learn by example how to walk, run, and jump within minutes to hours of birth - what is the data set they are using to learn these extremely complex motor skills, including the perception skills necessary to do so.
You are probably stuck on the idea of emergence. You imagine there must be some dividing line between intelligent and non intelligent. Therefore the point at which it emerges must be some spectacular miracle of engineering. When in fact there is no dividing line, just a continuum of consciousness from the small to the large. Read up on panspermia.
Whether GPT-3 can pass the Turing test or not doesn't prove that it possesses the same kind of intelligence as humans just that it can mimic the intelligence of humans. If you assume that they share a category because of this then you were probably already convinced. Everyone else is going to need a little more evidence.
We do know that the human brain can generalize from much less data.
> We do not know if the discrepancy is a matter of degree, or a matter of category.
That's not even a useful distinction. E.g. you could argue the difference between human intelligence and a random classifier is also a matter of degree.
I would say the difference is a matter of degree, but the difference right now is so enormous that it seems like a different category.
Adult human brains receive magnitudes more data than GPT-3 by age 18. Probably even by age one. Take vision, for example, which is believed to be approx. 8000x8000 24Hz stream with HDR (so more than 3 bytes per pixel). This alone generates (uncompressed) 252TB per year. Slightly over half of GPT-3 training data is Common Crawl, which only recently grew to 320TB.
Where have you seen a 2yo baby, who can solve tasks GPT-3 can?
If we fed a few years worth of video an audio to a neural network, could it write War and Peace? Could it play chess based on a text description of the game?
When you said "less data", you did not specify the type of data.
> Could it play chess based on a text description of the game?
We don't know - nobody tried it yet. We tried with 320TB text only.
> If we fed a few years worth of video an audio to a neural network, could it write War and Peace?
As Sonny from "I, robot" said, "can you?"
Obviously, it has to be the same data to make a comparison.
> We don't know - nobody tried it yet.
How would you even train a network unsupervised so that it can play chess from a text description? Nobody knows. That's the point.
>"can you?"
That doesn't matter, Tolstoy could, therefore it's within human capability.
No, it is very much not obvious. There's quite a bit of research showing models, that receive X samples of type A and Y samples on type B might be better in tasks on A, than models that are trained just on X.
There is no GPT-3 cognition. Spewing out readable but essentially meaningless random text is nothing at all like cognition.