Hasn't the typical human taken in orders of magnitude more data than this example? And the data has been of both direct sensory experience and texts from other people as well.
Have you read GPT-3s 175 billion parameters (words, sentences, papers, I don't care) of anything? Do you know all the words used in that corpus? Nobody has or does.
A child of a small age can listen to a very small set of things and not just come up with words to communicate with mama and papa what they learned, but they can reuse it. And this I think is key, because the language part of that is at least partially secondarily. The little kid understands what they're talking about even if they have a hard time communicating it to an adult. The fact they take creative leaps to use their extremely limited vocabulary to communicate their knowledge is amazing.
The human brain requires less training, but to some extent it is pretrained by our genetic code. The human brain will take on a predictable structure with any sort of training.
This post was generated using GPT-3. [;)]
Can’t tell if you are kidding or not, but if you aren’t, mind sharing links about the researcher for the curious?
edit: Can't seem to find it which is a shame. I think it may have been included in a TED talk.
Your post was generated using GPT-3 and 175 billion parameters of pre-existing human writing, contextualized, distilled, and cross-referenced with terminology we've agreed on for centuries. It's a parrot, and I remain unimpressed.
Take the learned knowledge of GPT-3 (because it must be so smart right?) and have it actually do something. Buy stocks, make chemical formulas, build planes. If you are not broke or dead by the end of that exercise, I'll be impressed and believe GPT-3 knows things.
So basically like DNA?
All DNA does it encode for how to grow, build, and mantain a human body. That human body has the potential to learn a language and communicate, but if you put a baby human inside an empty room and drop in food, it will never learn language and never communicate. DNA isn't magic and comparing "millions of years of evolution" of DNA is nothing like the Petabytes of data that GPT-3 needs to operate.
Again DNA has no knowledge embedded in it, it has no words or data embedded. Data in the sense that we imagine Wikipedia stored in JSON files on a hard disk. DNA stores an algorithm for growth of a human, that's it.
The GPT-3 model is probably > 700GB in size. That is, for GPT to be able to generate text it needs an absolutely massive "memory" of existing text which it can recite verbatim. In contrast, young human children can generate more novel insights with many orders of magnitude less data in "memory" and less training time.
Sigh. When DNA becomes human, it doesn't have a-priori access to all the world's knowledge and yet it still develops intelligence without it. And that little DNA machine learns and grows over time.
When thousands of scientists and billions of human artifacts and 1000X more compute are put into the philosophical successor of GPT-3, it won't be as impressive as what happens when a 2 year old becomes a 3 year old. (It will probably make GPT-4 even less impressive than GPT-3, because the inputs vis-a-vis outputs will be even that much more removed from what humans already do.)
If you mean that anything except full general intelligence is unimpressive than that seems like a fairly high standard.
What's unimpressive about a stunningly believable parrot? I think, at the very least, that GPT-3 is knowledgeable enough to answer any trivia you throw at it, and creative enough to write original poetry that a college student could have plausibly written.
Not everything worth doing is as high-stakes as buying stocks, making chemical formulas, or building planes.
> While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples [0]
I don’t know what constitutes an example in this case but let’s assume it means 1 blog article. I don’t know many humans that read thousands or tens of thousands of blog articles on a specific topic. And if I did I’d expect that human to write a much more interesting article.
To me, this and other similar generated texts from OpenAI feel bland / generic.
Take a listen to the generated music from OpenAI - https://openai.com/blog/jukebox/. It’s pretty bad, but in a weird way. It’s technically correct - in key, on beat, ect. And even some of the music it generates is technically hard to do, but it sounds so painfully generic.
> All the impressive achievements of deep learning amount to just curve fitting Judea Perl [1]
This comment was written by a human :)
[0]https://arxiv.org/abs/2005.14165 [1]https://www.quantamagazine.org/to-build-truly-intelligent-ma...
I'd like to play devils advocate here.
Given one blog article in a foreign language: Would a human be able to write coherent future articles?
With no teacher or context whatsoever how many articles would one have to read before they could write something that would 'fool' a native speaker? 1000, 100,000?
I have no idea how to measure the quantity/quality of contextual and sensory data we are constantly processing from just existing in the real world, however, it is vital to solving these tasks in a human way - yet it is a dataset that no machine has access to
I would argue comparing 'like for like' disregards the rich data we swim amongst as humans, making it an unfair comparison
Why then, the continued obsession with building single-media models?
Is focusing on the Turing test and language proficiency bringing us further away from the goals of legitimate intelligence?
I would argue "yes", which was my original comment. At no point in us trying to replicate what an adult sounds like have we actually demonstrated anything remotely like the IQ of a small child. And there's this big gap where it's implied by some that this process goes 1) sound like an adult -> 2) think like an adult, which seems to be missing the boat imo. (There's logically this intermediate step where we have this adult-sounding monster AI child.)
If we could constrain the vocabulary to that a child might be exposed to, the correlative trickery of these models would be more obvious. The (exceptionally good) quality of these curve fits wouldn't trick us with vocabulary and syntax that looks like something we'd say. The dumb things would sound dumb, and the smart things would sound smart. And maybe, probably even, that would require us fusing in all sorts of other experiential models to make that happen.
I think it's literally just working with available data. With some back of the envelope math, GPT-3's training corpus is thousands of lifetimes of language heard. All else equal, I'm sure the ML community would almost unanimously agree that thousands of lifetimes of other data with many modes of interaction and different media would be better. It would take forever to do and would cost insane amounts of money. But some kinds of labels are relatively cheap, and some data don't need labels at all, like this internet text corpus. I think that explains the obsession with single-media models. There's a lot more work to do and this is, believe it or not, still the low hanging fruit.
But why not just 1 lifetime of different kinds of data? Heck, why not an environment of 3 years of multi-media data that a child would experience? That wouldn't cost insane amounts of money (or probably anything even close to what we've spent on deep learning as a species).
A corpus limited to the experiences of a single agent would create a very compelling case for intelligence if at the end of that training there was something that sounded and acted smart. It couldn't "jump the gun" as it were, by a lookup of some very intelligent statement that was made somewhere else. It would imply the agent was creatively generating new models as opposed to finding pre-existing ones. It'd even be generous to plain-ol'-AI as well as deep learning, because it would allow both causal models to explain learned explicit knowledge (symbolic), or interesting tacit behavior (empirical ML).
How would you imagine creating such an environment in a way that allows you to train models quickly?