It learns new ideas with very few examples, to the extent you can express new ideas with a vocabulary of eight words.
English speakers tend to have a vocabulary of 30k words or so depending on what exactly you measure.
GPT-4 has 1.7 trillion parameters.
Of course scaling up is quite unpredictable in what capabilities it gets you, but It's not that much of a stretch to imagine that a GPT-4 sized model would have a reasonably sized vocabulary. Certainly worth testing if you have the resources to train such a thing!
Let's start with working vocabulary. Working vocabulary doesn't just mean knowing n words. It means putting n words together in factorially many valid combinations to construct sentences. And 30k btw is an insane working vocabulary. Most people know 1000 words on average in English. All of their sentences are structured from those 1000 words. This is true for most cases, except certain ones like Mandarin or German, where basic words can be used to assemble more complex words.
Certainly GPT-4 knows something. Presumably that something can be mapped to a working vocabulary. How large a vocabulary that is requires a testable, reproducible hypothesis supported by experimental proof. Do you have such a hypothesis with proof? Does anyone? Until we do it's just a guess.
Maybe I misunderstand but that sounds so stupid and wrong I don't know where to start. Standard vocab estimates are 15K - 30K, averaging about 20K (these from memory)
Edit: from wiki https://en.wikipedia.org/wiki/Vocabulary#Native-language_voc...
---
As a result, estimates vary from 10,000-17,000 word families[16][19] or 17,000-42,000 dictionary words for young adult native speakers of English.[12][17]
A 2016 study shows that 20-year-old English native speakers recognize on average 42,000 lemmas, ranging from 27,100 for the lowest 5% of the population to 51,700 lemmas for the highest 5%. These lemmas come from 6,100 word families in the lowest 5% of the population and 14,900 word families in the highest 5%. 60-year-olds know on average 6,000 lemmas more. [12]
According to another, earlier 1995 study junior-high students would be able to recognize the meanings of about 10,000–12,000 words, whereas for college students this number grows up to about 12,000–17,000 and for elderly adults up to about 17,000 or more.[20]
---
estimates i've heard for training gpt4 was $500m all in for what its worth
There was a post on here a few months ago about training using single characters as tokens instead of words, and it worked really well, being able to create new Shakespeare-like text despite not using human words as tokens. What a (human) word is can be learned by the model instead of encoded in the training set.
In logic there's also SKI combinator calculus, which is Turing-complete with three symbols, and unlike the Morse or A-Z examples, but like the Toki Pona example, each symbol has a semantic meaning.
If you just want to describe ideas in an abstract realm like sequences of colors, like this paper, it's not surprising you don't need many words.