It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.
It's so easy to downplay every advanced tech, it's actually fun.
Planes? Just a flying metal tube.
Self landing rockets? Just applied physics.
Smartphones? Just really good fab processes.
The internet? Just a bunch of computers.
CRISPR? Just a molecular scissor.
> GPT-3 is a really impressive milestone towards AGI
We really don't know this. It's a big step for the field of language models, that's for sure. But we're so far from AGI that nobody knows which direction it's in and whether it exists at all.
If a language model were able to do this task perfectly, it would be indistinguishable from intelligence, because continuing a sequence of words requires reasoning. You cannot conclude that has no understanding based solely on what it is trained to do when the task it is trained on would be sufficient to demonstrate understanding were it to fully succeed. There are lots of reasons to be skeptical of its potential, but this isn't one of them.
agreed.
but that's not everything we're basing our conclusions on – we also know that GPT-3 was trained purely on text, and i (and presumably GP) don't think that's a path towards "understanding".
in other words, i think being a language model [trained only using a text corpus] is a valid reason to be skeptical of its potential :)
1. It's all just data in the end. The signals from your eye are not any more real because they are in response to photons.
2. Deafblind people exist, and can even learn to speak vocally. They might have misconceptions about the visual world, but it's certainly not true that they lack general intelligence.
3. As an objection to GPT-3 as a pathway to AGI, the only thing stopping us training these models on high-bandwidth highly-multimodal data is scale, anyway, so the objection doesn't work.
4. GPT-3 is clearly capable of reasoning in ways that can only be explained with more sophisticated world models than at smaller scales. People asserted reasoning was at a limit with GPT-2, and it just clearly isn't.
What does this mean? The world with countries, borders, celebrities, hypes, newsletters and ideologies?
Because that world is as virtual and not grounded in reality as the world of data points fed into this model. Culture is made up. Language is made up. It may express itself in reality but so do the algorithms of social network sites.
If something can only live in a reality of data points on the internet this thing would live mostly in the same reality as we do.
We don't know enough about human cognition to say this.
Scott Aaronson has something interesting to say about this in a conversation with Lex Fridman, actually: https://youtu.be/G_-BBniFFCM?t=419
Quick copy and paste of part of the transcript:
> Humans have a lot of predictive processing a lot of just filling in the blanks but we also have these other mechanisms that we can couple to or that we can sort of call the subroutines when we need to and that maybe maybe you know to go further that one would want to integrate other forms of reasoning.
No you're not, and that is very easy to disprove. Look at the sentence "John took the water bottle out of the backpack so that it would be lighter". What does it refer to in the sentence, the bottle or the backpack?
Did it statistically come to you or did you need to consult Google? No, you know the right answer, it's the backpack. Why? Because you have a physical understanding of the world. The bottle doesn't get lighter, when you take it out of the backpack, the backpack does, because the bottle is not in there any more. This is not statistics, it's not manipulating strings, it's having a fundamental physical model of the world in your head, and an idea about how entities operate in it.
When you talk you don't do random statistical inference, you match language to the semantics you want to express, which is not statistical.
let lo = 0;
let hi = vec.len();
let mid = lo + (hi
GPT is not trying to make a point and is not capable of changing its mind. You, hopefully, are.
Edit: I don't think you should be getting downvoted because it's a valid (and interesting) question.
But this uncovers difficult questions about free will. If we're all just autocompleting based on a combination of the world around us, our internal state, and the physical laws, then what even is intelligence anyway? This view reduces thought to nothing more than an interesting dust storm.
Still, I find the original argument compelling, if not logically convincing. There does seem to be something missing from GPT-3 that differs fundamentally from human intelligence or AGI. But maybe that's an illusion.
Edit: I don't think you should have been downvoted, since your question is valid and constructive in my view.
That's pretty much it. I do belive it's possible to actually develop "will", but almost nobody thinks that they need to work on such things. They confuse being a programmed robot with being a programmer.
It may be, but there's a lot in that fancy. If it were 'just' an autocomplete we'd all be using markov chains for our dumb chatbots like we were in the 2000s
That said, I agree more closely with LeCun than the hypers here.