Hacker News Reader

Comment by qolop | Hacker News Reader

>> It's still just predicting the next word

Predicting the next word is a much deeper problem than people like you realise. To be able to be good at predicting the next word you need to have an internal model of the reality that produced that next word.

GPT-4 might be trained at predicting the next word, but in that process it learns a very deep representation of our world. That explains how it has an intuition for colours despite never having seen colours. It explains why it knows how physical objects in the real world interact.

Now, if you disagree with this hypothesis it's very easy to disprove it by presenting a problem to GPT4 that is very easy for humans to solve but not for GPT4. Like the Yann Lecun gear problem, which GPT4 is also able to solve.

View on HN

“To be able to be good at predicting the next word you need to have an internal model of the reality that produced that next word.”

Now that’s an interesting claim - that I would deeply dispute. It learns from text. Text itself is a model of reality. So chatgtp if anything proves that in order to be good at predicting the next word all you need is a good model of a model of reality. GTP knows nothing of actual reality only the statistics around symbol patterns that occur in text.

You are being given a chance to dispute it. Give an example of a problem that any human would be easily able to solve but GPT4 wouldn't.

>> "good model of a model of reality"

That is just a model of reality. Also, a "model of reality" is what you'd typically call a world model. Its an intuition for how the world works, how people behave, that apples fall from trees and that orange is more similar to red than it is to grey.

Your last line shows that you still have a superficial understanding of what its learning. Yes it is statistics, but even our understanding of the world is statistical. The equations we have in our head of how the world works are not exact, they're probabilistic. Humans know that "Apples fall from the _____" should be filled with 'tree' with a high probability because that's where apples grow. Yes, we have seen them grow there, whereas the AI model has only read about the growing on trees. But that distinction is moot because both the AI model and humans express their understanding in the same way. The assertion we're making is that to be able to predict the next word well, you need an internal world model. And GPT4 has learnt that world model well, despite not having sensory inputs.

Can chatgtp ride a bicycle? Can you ride a bicycle? If you ‘d never rode on a bicycle before - do you think if you read enough books on bicycle riding, the physics of bicycle riding, the physics of the universe - you would have anywhere near as complete a model of bicycle riding as someone who’d actually rode on a bicycle before. Sure you’d be able to talk a great game about riding bicycles - but when it comes to the crunch, you’d fall flat on your face. That’s because riding a bicycle involves a large number of incredibly complex emergent control phenomena embedded within the marvel of engineering that is the human body - not just the small part of the brain that handles language. So call me when LLM’s can convert their ‘world models’ learned from statistics on human language use into being able to ride a bicycle first time. Until then I feel comfortable in the knowledge they know virtually nothing of our objective reality.

Could Stephen Hawking ride a bicycle?

Yes, his mnd was diagnosed around the age of 21? And he didn’t learn to ride bicycles from reading books.

Reply on news.ycombinator.com