Comment by bccdee | Hacker News Reader

Comment by bccdee | Hacker News Reader

Parent Full thread

> So people are going to be even more blindsided when someone develops an AI that uses GPT as its language comprehension component

I don't think that would work, because GPT doesn't actually comprehend anything. Comprehension requires deriving meaning, and GPT doesn't engage with meaning at all. It predicts which word is most likely to come next in a sequence, but that's it.

What I think we'd be more likely to end up with is something GPT-esque which, instead of simply generating text, transforms English to and from a symbolic logic language. This logic language would be able to encode actual knowledge and ideas, and it would be used by a separate, problem-solving AI which is capable of true logic and analysis—a true general AI.

The real question, IMO, is if we're even capable of producing enough training data to take such a problem-solving AI to a serious level of intelligence. Scenarios that require genuine intelligence to solve likely require genuine intelligence to create, and we'd need a lot of them.

>Comprehension requires deriving meaning, and GPT doesn't engage with meaning at all. It predicts which word is most likely to come next in a sequence, but that's it.

Why think that "engaging with meaning" is not in the solution-space of predicting the next token? What concept of meaning are you using?

I get what you mean here but they probably mean referential meaning... having never seen a dog, GPT doesn't really know what a dog is on a physical level, just how that word relates to other words.

How do blind people know what a dog is?

Probably by hearing, touch, etc. - my point is some stimulus from reality, doesn't have to be any of our senses, just some stimulus.

And the more stimuli, and the more high resolution and detailed (say at the atomic level), the more GPT's model of reality would be accurate.

Language is just symbols that stand for a stimulus (in the best case)

You could argue that GPT has a model of meaning somewhere inside of it, but that's besides the point. If that meaning is hiding latent inside GPT, then it's not accessible to any other system which might want to use GPT as an interface. GPT accepts English as input and produces English as output; that's it.

That said, no, I don't think GPT properly grasps meaning, and my reason for that is simple: It regularly contradicts itself. It can put together words in a meaningful-looking order, but if it actually understood what those words meant as an emergent property of its design, then you wouldn't be able to trick it into saying things that don't make sense or are contradictory. If someone actually understands a subject, they won't make obvious mistakes when they discuss it; since GPT makes obvious mistakes, it can't actually grasp meaning—only brush up against it.

>since GPT makes obvious mistakes, it can't actually grasp meaning—only brush up against it.

This argument doesn't apply to GPT because it isn't a single coherent entity with a single source of truth. GPT is more like a collection of personas, which persona you get is determined by how you query it. One persona may say things that contradict other personas. Even within personas you may get contradictory statements because global consistency is not a feature that improves training performance, and can even hinder it. Samples from its training data are expected to be inconsistent.

It is important not to uncritically project expectations onto LLMs derived from our experiences with human agents. Their architecture and training regime is vastly different than humans and so we should expect their abilities to manifest differently than analogous abilities in humans. We can be easily mislead if we don't modify our expectations for this alien context.

I think if you could somehow examine the output of your language model in isolation, you would find it also doesn't "comprehend". Comprehension is what we assign to our higher level cognitive models. It is difficult to introspectively isolate your own language center, though.

I took a stab at an exercise that may allow you to witness this within your own mind here: https://www.jerf.org/iri/post/2023/streampocalypse-and-first... Don't know if it works for anyone but me, of course, but it's at least an attempt at it.

I think you put the wrong link? https://www.jerf.org/iri/post/2023/understanding_gpt_better/ maybe?

Yes, you are correct. Oops. Too late to correct.

The language centres of our brain don't know what a dog is, but they can take the word "dog" and express it on a level that the logic centres of our brain can use. I don't know if "comprehending" is the right word, exactly, but it's transforming information from one medium to another in preparation for semantic and logical analysis.

GPT doesn't do that. What it does is related to meaning, but unlike the language comprehension parts of our brains, which are (presumably) stepping stones between language and reason, GPT doesn't connect to any reasoning thing. It can't. It's not built to interface with anything like that. It just reproduces patterns in language rather than extracting semantic meaning from them in a way that another system can use. I'm not saying that's more or less complicated—just different.

Reply on news.ycombinator.com