Hacker News Reader

Comment by smeeth | Hacker News Reader

Why must we keep having this argument?

If you do research in the field you know full well that GPT/any other transformer or Bert model is generating text by regurgitating approximate conditional probabilities of words given all the text it has ever seen and the prompt. The neurophysiological concept of “understanding” as most understand it is orthogonal to the way the algorithm actually works.

A more useful conversation to have might be: what sort of prompts does GPT struggle with? How might we alter the algorithm to ameliorate these issues? But instead we separate into cults of believers and nonbelievers and uselessly wax poetic about it.

View on HN

> If you do research in the field you

The hype machine is full-on marketing GPT-3 and promised solutions based on it to normal people, so "but researchers know this" is not enough.

> The neurophysiological concept of “understanding” as most understand it is orthogonal to the way the algorithm actually works.

This is not obviously true and it's exactly the core of the debate. A GPT-3 proponent might say: We don't really know what "understanding" means, so it very well might be nothing more than complex rehashing of conditional probabilities. This isn't implausible. Consider Friston's "free energy principle" which leads to the conclusion that brain function is determined entirely by prediction.

That’s a good point and thanks for the reference.

I added “as most understand it” to caveat cases like this one, where there exists a non-falsifiable theory about how cognition works under which the GPT algorithm and “understanding” would be non-orthogonal.

Don’t get me wrong, it’s an interesting theory, but with no evidence of existence or non-existence do we really need to spend this much time on it? This is why I invoked cults - arguing about theories without evidence smells a more like a religious argument than a scientific one.

I think I mostly just wish we could end the argument by all agreeing the following (I think) non-controversial points...

1) GPT is very impressive 2) GPT is not perfect 3) we don’t have a fucking clue how human cognition works 4) because of 3, how “close” GPT is to human cognition is an open question

> Why must we keep having this argument?

The fact that GPT-3 is such an impressive leap in regurgitation ability means that many more people are going to be hearing about it and it will be used in many more contexts.

> If you do research in the field

With it approaching a cusp of mainstream use it's becoming more important than ever for people (everywhere, not just in tech) to understand what it is and isn't.

There are going to be people who see an impressive curated sample and believe GPT-3 is almost a person. That doesn't help anything.

There are already several comments here that put the word "understanding" in quotation marks or italics. It is beginning to be used in the same way that "consciousness" used to be used, as a kind of ill-defined catch-all for something that separates humans from machines.

Yes, there are clearly failures in reasoning, binding, and coherence in many of the examples here. There are many other cases where it does ok with simple reasoning tasks, maintains cohesion over many paragraphs, and successfully creates formal or generic text such as poetry, code, stylistic imitation.

I don't think that everyone who does research in the field would agree with your comment, or the article. More and more often I see people saying "real researchers in the field" know that GPT-3 has no understanding or reasoning ability, but I know people researching in the field who disagree with that.

Because modern language models are good enough that the question may soon be directly relevant. If we invent a bot with reliable human-level conversational capability, that's going to have a huge impact on the real world beyond just its implications for further AI research. The fact that "understanding" is orthogonal to the mechanics of the program makes the question all the more concerning, because it raises the likelihood that some minor change could leapfrog a model from "kinda reasonable but says dumb things a lot" to some functional equivalent of human understanding.

> regurgitating approximate conditional probabilities of words given all the text it has ever seen and the prompt

This is meaningless; you have only described the task. It is equally applicable to a superintelligence as it is of a Markov chain.

Sort of, my point was that human/superhuman intelligence is widely understood by the public to work differently than a markov chain.

GPT-3 also works differently to a Markov chain.

> A more useful conversation to have might be: what sort of prompts does GPT struggle with? How might we alter the algorithm to ameliorate these issues?

That would be eminently useful, but unfortunately we can't have that discussion because OpenAI aren't exposing the model.

They've really brought this on themselves - I don't think there'd be these believer/nonbeliever camps if they had taken the slower, rationalist/scientific approach to the research.

Instead, they've breathlessly hyped up their new API with media releases and saturated social media, and are picking and choosing who they allow to play with their model. It's not surprising that a lot of people didn't take too kindly to it.

Reply on news.ycombinator.com