I disagree with Geoff Hinton regarding "glorified autocomplete" | Hacker News Reader

I disagree with Geoff Hinton regarding "glorified autocomplete"

statmodeling.stat.columbia.edu

194 pointsmagoghm264 comments

I agree with Hinton, although a lot hinges on your definition of "understand."

I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go.

Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they found is that there isn't really a limit to how good it can get. There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train. And it can go far beyond what the best human Go player can ever do. Even though very smart humans have spent their lifetimes deeply studying the game, and Alpha Zero had to learn everything from scratch.

One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move. And that model is obviously very sophisticated, and if you are going to say that it doesn't understand the game of Go, I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.

Likewise, with large language models, while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead. As Hinton says, there is no real limit to how sophisticated they can get. When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains. And the closer it gets, the more sophisticated model it needs. In the sense that Alpha Zero needs to "understand" the game of Go to play effectively, the large language model needs to understand "the world" to get better at predicting.

The difference is that "the world" is not exhaustible in the same way as Go is. While it's surely true that the number of possible overall Go game states is extremely large, the game itself is trivially representable as a set of legal moves and rules. The "world model" of the Go board is actually just already exhaustive and finite, and the computer's work in playing against itself is to generate more varied data within that model rather than to develop that model itself. We know that when Alpha Zero plays a game against itself it is valuable data because it is a legitimate game which most likely represents a new situation it hasn't seen before and thus expands its capacity.

For an LLM, this is not even close to being the case. The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. And we certainly haven't figured out a "model" which would let a computer generate new and valid data that expands its understanding of the world beyond its inputs, so self-training is a non-starter for LLMs. What the LLM is "understanding", and what it is reinforced to "understand" is not the world but the format of texts, and while it may get very good at understanding the format of texts, that isn't equivalent to an understanding of the world.

>The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility.

No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts.

To the bird who can intuitively sense and use electromagnetic waves for motion and guidance, your model of the world is fundamentally incomplete.

There is a projection of the world in text. Moreover training on additional modalities is trivial for a transformer. That's all that matters.

That's the difference though. I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. To a neural network there is no distinction. The closest the entire dynamic comes is the very basic annotation of RLHF which itself is done by an external human who is providing the value judgment, but even that is absent once training is over.

Despite not having the bird's sense for electromagnetic waves, I have an understanding that they are there, because humans saw behavior they couldn't describe and investigated, in a back-and-forth with a world that has some capacity to disprove hypotheses.

Additional modalities are really just reducible to more kinds of text. That still doesn't exhaust the world, and unless a machine has some ability to integrate new data in real time alongside a meaningful commitment and accountability to the world as a world, it won't be able to cope with the real world in a way that would constitute genuine intelligence.

>I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins.

Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception.

And yes LLMs can in fact tell truth from fiction.

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

Your argument seems to boil down to "they can't perform experiments" but that isn't true either.

It is a very basic fact that LLMs have no concept of true or false, it only has an ability to look up what text data it has seen before. If you do not understand this you are in no position to discuss LLMs.

I really don't know what people mean when they say this. We routinely instruct computer chips to evaluate whether some condition is true and take action on that basis, even though the chip is "just" a selectively doped rock. Why would the details of an LLM's underlying architecture mean that it can't have a concept of true or false?

One of the most ridiculous comments I have read about LLMs here.

The ~100 layer deep neural networks infer many levels of features over the text, including the concept of true and false. That is trivial for an LLM.

Are you completely unaware these are based on deep neural networks?

Convolutional Neural Networks don't operate by "look up" of text data.

Okay, so then tell me how does it decide whether it is true or false that Biden is the POTUS?

It's response is not based on facts about the world as it exists, but on the text data it has been trained on. As such, it is not able to determine true or false even if the response in the above example would be correct.

Serious question, in pursuit of understanding where you're coming from: in what way do you think that your own reckoning is fundamentally different to or more "real" than what you're describing above?

I know I don't experience the world as it is, but rather through a whole bunch of different signals I get that give me some hints about what the real world might be. For example, text.

You understand the concept of true vs false.

LLM does not, that isn't how it works.

You can say the difference is academic but there is a difference.

What is the difference between a real good faker of intelligence and actual intelligence is an open question.

But I will say most AI experts agree that LLM are not artificial general intelligence. It isn't just a lack of training data, they just are not of the category that we mean by that.

> You understand the concept of true vs false.

> LLM does not, that isn't how it works.

GPT-4 can explain the concept when prompted and can evaluate logic problems better than most human beings can. I would say it has a deeper understanding of "true vs false" than most humans.

I think what you are trying to say is that LLMs are not conscious. Consciousness has no precise universally agreed formal definition, but we all know that LLMs are not conscious.

> GPT-4 can explain the concept when prompted and can evaluate logic problems better than most human beings can. I would say it has a deeper understanding of "true vs false" than most humans.

Sigh

GPT produces output which obeys the patterns it has been trained on for definitions of true and false. It does not understand anything. It is a token manipulation machine. It does it well enough that it convinces you, a walking ape, that it understands. It does not.

A human is an ape that is obeying patterns that it has been trained on. What is school but a bunch of apes being trained to obey patterns? Some of these apes do well enough to convince you that it understands things. Some apes fully "understand" that flat earth theory is true, or they "understand" that the Apollo moon landings were faked.

You have a subjective philosophical disagreement about what constitutes understanding. That is fine. I clearly understand it is not conscious and that programs do not understand things the way that humans do. We are fundamentally different to LLMs. That is obvious. But you are not making a technical argument here unless you can define "understand" in technical terms. This is a matter of semantics.

> It is a token manipulation machine

Deep learning and machine learning in general is more than token manipulation. They are designed for pattern recognition.

You acknowledged above that consciousness isn't what LLM is and you likely understand that the poster was referring to that...

The broad strokes you use here are exactly why discussing LLMs are hard. Sure some people dismiss them because it isn't general AI but having supporters dismiss any argument with "passes the Turning test" is equally useless.

No you have misunderstood. As I wrote above:

"But you are not making a technical argument here unless you can define "understand" in technical terms. This is a matter of semantics."

I said the nature of their argument is not technical, since they are not dealing with technical definitions, but I did not dismiss their argument altogether. I clarified and restated their own argument for them in clearer terms. LLMs are not conscious, but they can still "understand" very well depending on your definition of understand. Understanding is not a synonym for consciousness. Language is evolving and you need to be more precise when discussing AI / machine learning.

One definition of understand is:

"perceive the intended meaning of (words, a language, or a speaker)."

Deep learning models recognize patterns. Mechanical perception of patterns. They understand things mechanically, unconsciously.

I stand by my point that people using synonyms for consciousness being told "LLM knows true better than humans do" is bad for discussion.

The core issue is their "knowledge" is too context sensitive.

Certainly humans are very context sensitive in our memories but we all have something akin to a "mental model" we can use to find things without that context.

In contrast LLM has knowledge defined by that context quite literally.

In either case my original point on using true and false is that LLM can hallucinate and on a fundamental design level there is little that can be done to stop it.

LLMs can outperform humans on a variety of NLP tasks that require understanding. Formally, they are designed to solve "natural language understanding" tasks as a subset of "natural language processing" tasks. The word "understanding" is used in the academic context here. It is a standard term in NLP research.

https://en.wikipedia.org/wiki/Natural-language_understanding

My point was to show that their thinking, reasoning and language was flawed, that it lacked nuance and rigor. I am trying to raise the standards of discussion. They need to think more deeply about what "understanding" really means. Consciousness does not even have a formal universally agreed definition.

Sloppy non-rigorous shallow arguments are bad for discussion.

> LLM can hallucinate and on a fundamental design level there is little that can be done to stop it.

That's a separate issue. They generally don't hallucinate when solving a problem within their context window. Recalling facts from their training set is another issue.

Humans sometimes have a similar problem of "hallucinating" when recalling facts from their long term memory.

Except that if you narrow to a tiny training set you are back to problems that can be solved almost as quickly with full text search...

Narrow to a tiny training set? What are you talking about now? That has nothing to do with deep learning.

GPT-3.5 was trained on at least 300 billion tokens. It has 96 layers in its neural network of 175 billion parameters. Each one of those 96 stacked layers has an attention mechanism that recomputes an attention score for every token in the context window, for each new token generated in sequence. GPT-4 is much bigger than that. The scale and complexity of these models is beyond comprehension. We're talking about LLMs, not SLMs.

I misread context window as training set and thought you were switching to SLMs. My mistake.

In order to affirm something is true, you don't just need to know it, you need to know that you know it. LLMs fundamentally have no self-knowledge.

> LLMs fundamentally have no self-knowledge

ChatGPT can tell me about itself when prompted. It tells me that it is an LLM. It can tell me about capabilities and limitations. It can describe the algorithms that generate itself. It has deep self knowledge, but is not conscious.

LLMs only knows it's text embeddings. It does not know the real world. Clear?

Humans and other creatures only know their sensory data input. Therefore they also don't know the real world.

Your eyes and ears perceive a tiny minuscule fraction of what is out there in the real world.

A blind and deaf person must know even less of the real world than an LLM, which can read more than a human can ever read in their lifetime.

It’s giving the most likely answer as opposed to the factual answer?

> It's response is not based on facts about the world as it exists, but on the text data it has been trained on

How did you find out that Biden was elected if not through language by reading or listening to news? Do you have extra sensory perception? Psychic powers? Do you magically perceive "facts" without any sensory input or communication? Ridiculous.

By the same argument your knowledge is also not based on "facts" about the world, since you only learned about it by reading or listening. Absurd nonsense.

You didn't answer my question ergo you concede that LLMs don't know true or false.

I did answer your question indirectly. By the reasoning in your argument, you yourself also don't know true or false. Your argument is logically flawed.

Do LLMs know true or false? It depends on how you define "know". By some definitions, they "know true or false" better than humans, as they can explain the concept and solve logic problems better than most humans can. However, by any definition that requires consciousness, they do not know because they are not conscious.

The average person spends a lot of time completely immersed in "false" entertainment. Actors are all liars, pretending to be someone they are not, doing things that didn't really happen, and yet many people are convinced it is all "true" for at least a few minutes.

People also believe crazy things like Flat Earth theory or that the Apollo moon landings were faked.

So LLMs have a conceptual understanding of true/false, strong logical problem solving to evaluate truth or falsity of logical statements, and factual understanding of what is true and false, better than many humans do. But they are not conscious therefore they are not conscious of what is true or false.

It certainly doesn't "look up" text data it has seen before. That shows a fundamental misunderstanding of how this stuff works. That's exactly why I use the example above of Alpha Zero and how it learns to play Go, since that demonstrates very clearly that it's not just looking things up.

And I have no idea what you mean by saying that it has no concept of true or false. Even the simplest computer programs have a concept of true or false, that's kind of the simplest data type, a boolean. Large language models have a much more sophisticated concept of true and false that has a lot more nuance. That's really a pretty ridiculous thing to say.

Yes, you don't understand what I said. The model has no concept of true or false. It only has embeddings. If 'asked' a question it can see if that is consistent with its embeddings and probabilities or not. This is not a representation of the real world, of facts, but simply a product of its training.

"This is not a representation of the real world, of facts, but simply a product of its training."

Tell me how that doesn't apply to the human brain as well.

They have no inherent concept of true or false, sure. But what are you comparing them to? It would be bold to propose that humans have some inherent concept of true or false in a way that LLMs do not; for both humans and LLMs it seems to be emergent.

In all these arguments its implied that this "genuine intelligence" is something humans all have, and nothing could be farther from the truth, that is why we have flat earthers or religious people and many other people beliving for decades easily refutable lies.

There is no such thing as a world model, and you don't have one of them. This is a leftover bad psychological concept from the 70s AI researchers who never got anywhere. People and other creatures do very little modeling things, they mostly just do stuff.

World model means inner representation of the external world. Any organism with a functioning brain has a world model. That's what brains do.

If you don't have a world model then you are a vegetable and could not be replying on HN.

If you close your eyes, how long can you navigate in the environment without hitting something? Not long, because you didn't model it.

If you're taking out the recycling, do you take the time to identify (model) each piece of it first? No, because that's not necessary.

Wait, you actually think we are talking about modelling as a conscious deliberate process in active working memory? Well there's your fundamental mistake. That is not what we are discussing, not even remotely.

The vast model in your brain is learned and generated unconsciously without your direct awareness.

No, I didn't say anything about doing it consciously. Motion is largely unconscious, like how you can throw things at a target without thinking about it.

But if you're just using it to mean "factual memory", calling it modeling seems like false precision.

Oh well in that case the answer is straightforward.

If you close your eyes and get lost after a few seconds, that's because that aspect of your model was not a 100% perfect exact replica of external reality that extended infinitely far in all spatial directions at all resolutions. For example, your internal spatial model is limited to some degree of accuracy and does not include the entire surface of Mars, but that doesn't mean that your model does not exist at all. Models are not perfect by definition. I thought this would be obvious.

Why would you think any model has to be a perfect exact 1:1 representation of the entire universe?

The model of reality in your head is a simplification that serves a purpose. Arbitrarily closing your fully functioning eyes is not something your model generating hardware was evolutionarily optimized for. Natural selection weeds out that kind of behaviour.

If you become blind then your model will change and optimize for other sensory inputs. Think of a blind man with a cane.

> For example, your internal spatial model is limited to some degree of accuracy and does not include the entire surface of Mars, but that doesn't mean that your model does not exist at all.

You're using "your model" as a metaphorical term here, but if you came up with any precise definition of the term here, it'd turn out to be wrong; people have tried this since the 50s and never gotten it correct. (For instance, is it actually a singular "a model" or is it different disconnected things you're using a single name for?)

See Phil Agre (1997) on exactly this idea: https://pages.gseis.ucla.edu/faculty/agre/critical.html

David Chapman (more general and current): https://metarationality.com/rationalism

and this guy was saying it in the 70s: https://en.wikipedia.org/wiki/Hubert_Dreyfus#Dreyfus'_critic...

> limited to some degree of accuracy

This isn't the only issue:

- You may not have observed something in the room in the right way for the action you need to do later.

- You might have observed it in a way you don't need later, which is a waste of time and energy.

- It might change while you're not looking.

- You might just forget it. (Since people do this, this must be an adaptive behavior - "natural selection" - but it's not a good thing in a model.)

> Why would you think any model has to be a perfect exact 1:1 representation of the entire universe?

What principle can you use to decide how precise it should be? (You can't do this; there isn't one.)

> The model of reality in your head is a simplification that serves a purpose.

Not only does it serve a purpose, your observations largely don't exist until you have a purpose for them.

RL agents tend to get stuck investigating irrelevant things when they try to maintain models; humans are built to actively avoid this with attention and boredom. Robot cameras take in their entire visual field and try to interpret it; humans both consciously and unconsciously actively investigate the environment as needed alongside deciding what to do. (Your vision is mostly fake; your eyes are rapidly moving around to update it only after you unconsciously pay attention to something.)

> Natural selection weeds out that kind of behaviour.

Not that well since something like half of Americans are myopic…

So basically you agree with what I was saying.

> What principle can you use to decide how precise it should be?

It is not up to me or anyone else to decide. Our subjective definitions and concepts of the model are irrelevant. How the brain works is a result of our genetic structure. We don't have a choice.

You can design a human if you want, that's what artificial intelligence is supposedly all about.

Anyway, read the paper I linked.

All of this was in response to your comment earlier:

"There is no such thing as a world model, and you don't have one of them."

There is such a thing as a world model in humans, and we all have them otherwise we could not think about or conceptualize or navigate the world. Then you have discussed how to define or construct a useful model or the limitations of a model but that is not relevant to the original point and I'm already aware of that.

I do agree, but more importantly love this part of the argument! Its when all the personality differences become too much to bear and suddenly people are accused of not even knowing themselves. Been there before, what a wild ride!

> suddenly people are accused of not even knowing themselves

It's not some desperate retort. People don't know themselves very well. Look at the research into confabulation, it seems to be standard operating procedure for human brains.

Kant would like a word with you about your point on whether people themselves understand the world and not just the format of their perceptions... :)

I think if you're going to be strict about this, you have to defend against the point of view that the same 'ding an sich' problem applies to both LLMs and people. And also whether if you had a limit sequence of KL divergences, one from a person's POV of the world, and one from an LLM's POV of texts, what it is about how a person approaches better grasp of reality - and likewise their KL divergence approaches 0, in some sense implying that their world model is becoming the same as the distribution of the world - that can only apply to people.

It seems possible to me that there is probably a great deal of lurking anthropocentrism that humanity is going to start noticing more and more in ourselves in the coming years, probably in both the direction of AI and the direction of other animals as we start to understand both better

The world on our plane of existence absolutely is exhaustible, just on a much, much larger scale. Doesn't mean that the process is fundamentally different, and for the human perspective there might be diminishing returns.

What if we are just the result of a ml network with a model of the world?

We're not.

LLMs are very good at uncovering the mathematical relationships between words, many layers deep. Calling that understanding is a claim about what understanding is. But because we know how the LLMs we're talking about at the moment are trained, it seems to have more problems:

LLMs do not directly model the world; they train on and model what people write about the world. It is an AI model of a computed gestalt human model of the world, rather than a model of the world directly. If you ask it a question, it tells you what it models someone else (a gestalt of human writing) is most likely say. That in turn is strengthened if user interaction accepts it and corrected only if someone tells it something different.

If we were to define that as what "understanding" is, we would equivalently be saying that a human bullshit artist would have expert understanding if only they produced more believable bullshit. (They also just "try to sound like an expert".)

Likewise, I'm not convinced that we can measure its understanding just by identifying inaccuracies or measuring the difference between its answers and expert answers - There would be no difference between bluffing your way through the interview (relying on your interviewer's limitations in how they interrogate you) and acing the interview.

There seems to be a fundamental difference in levels of indirection. Where we "map the territory", LLMs "map the maps of the territory".

It can be an arbitrarily good approximation, and practically very useful, but it's a strong ontological step to say one thing "is" another just because it can be used like it.

"LLMs do not directly model the world; they train on and model what people write about the world"

This is true. But human brains don't directly model the world either, they form an internal model based on what comes in through their senses. Humans have the advantage of being more "multi-modal," but that doesn't mean that they get more information or better information.

Much of my "modeling of the world" comes from the fact that I've read a lot of text. But of course I haven't read even a tiny fraction of what GPT4 has.

That said, LLMs can already train on images, as GPT4-V does. And the image generators as well do this, it's just a matter of time before the two are fully integrated. Later we'll see a lot more training on video and sound, and it all being integrated into a single model.

We could anthropomorphize any textbook too and claim it has human level understanding of the subject. We could then claim the second edition of the textbook understands the subject better than the first. Anyone who claims the LLM "understands" is doing exactly this. What makes the LLM more absurd though is the LLM will actually tell you it doesn't understand anything while a book remains silent but people want to pretend we are living in the Matrix and the LLM is alive.

Most arguments then descend into confusing the human knowledge embedded in a textbook with the human agency to apply the embedded knowledge. Software that extracts the knowledge from all textbooks has nothing to do with the human agency to use that knowledge.

I love chatGPT4 and had signed up in the first few hours it was released but I actually canceled my subscription yesterday. Part because of the bullshit with the company these past few days but also because it had just become a waste of time the past few months for me. I learned so much this year but I hit a wall that to make any progress I need to read the textbooks on the subjects I am interested in just like I had to this time last year before chatGPT.

We also shouldn't forget that children anthropomorphize toys and dolls quite naturally. It is entirely natural to anthropomorphize a LLM and especially when it is designed to pretend it is typing back a response like a human would. It is not bullshitting you though when it pretends to type back a response about how it doesn't actually understand what it is writing.

> One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move.

It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move (or moves that are better than others) and hence a function that goes from the state of the game to the best move. We just don't have a way to compute this function, but it exists. And that function doesn't need the concept of lookahead, that's just an intuitive way of how could find some of its values. Likewise ML algorithms don't necessarily need lookahead, they can just try to approximate that function with enough precision by exploiting patterns in it. And that's why we can still craft puzzles that some AIs can't solve but humans can, by exploiting edge cases in that function that the ML algorithm didn't notice but are solvable with understanding of the game.

The thing is though, does this really matter if eventually we won't be able to notice the difference?

> It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move

Is there really a difference between the two? If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"? In other words, when humans strategize and predict what happens in the real world, aren't they doing the same thing?

I suppose you could argue that humans also include additional world models in their planning, but it's not clear to me that these models are missing and impossible for machine learning models to generate during training.

> If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"?

You're confusing the reason why a move is good with how you can find that move. Yeah, a move is good due to how it shapes the opponent remaining moves, and this is also the reasoning we make in order to find that move, but it doesn't mean you can only find that move by doing that reasoning. You could have found that move just by randomly picking one, it's not very probably but it's possible. AIs just try to maximize such probability of picking a good move, meanwhile we try to find a reason a move is good. IMO it doesn't make sense to try to fit the way AI do this into our mental model, since the middle goal is fundamentally different.

> Since Go is a deterministic game there is always a best move

The rules of the game are deterministic, but you may be going a step too far with that claim.

Is the game deterministic when your opponent is non-deterministic?

Is there an optimal move for any board state given that various opponents have varying strategies? What may be the best move against one opponent may not be the best move against another opponent.

Maybe "deterministic" is not the correct term here. What I meant is that there's no probability or unknown in the game, so you can always know what are the possible moves and the relative new state.

The opponent's moves may be considered non-deterministic, but you can just assume the worst case for you, that is the best case for the opponent, which is the opponent will always play the best move too.

At every point in time there are a range of moves with different levels of optimality. That range changes at the next point in time following the opponent's move.

The opponents strategy is an unknown variable not determined by the current board state.

Therefore the best move cannot be determined by the current board state, as it cannot be determined in isolation from the opponents strategy.

The optimal strategy can be determined from the current state. This is the principle behind minimax.

In a perfect information zero sum game, we can theoretically draw a complete game tree, each terminal node ending with a win, loss, or draw. With a full understanding of the game tree we can make moves to minimize our opponent’s best move.

I stand corrected. Thanks for that explanation.

> to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move.

While I imagine alpha go does some brute force and some tree exploration, I think the main "intelligent" component of alpha go is the ability to recognize a "good" game state from a "bad" game state based on that moment in time, not any future plans or possibilities. That pattern recognition is all it has once its planning algorithm has reached the leaves of the trees. Correct me if I'm wrong, but I doubt alpha go has a neural net evaluating an entire tree of moves all at once to discover meta strategies like "the opponent focusing on this area" or "the opponent feeling on the back foot."

You can therefore imagine a pattern recognition algorithm so good that it is able to pick a move by only looking 1 move into the future, based solely on local stone densities and structures. Just play wherever improves the board state the most. It does not even need to "understand" that a game is being played.

> while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead.

So I don't think this statement is necessarily true. "Understanding" is a major achievement, but I don't think it requires planning. A computer can understand that 2+2=4 or where to play in tic-tac-toe without any "planning".

That said, there's probably not much special about the concept of planning either. If it's just simulating a tree of future possibilities and pruning it based on evaluation, then many algorithms have already achieved that.

The "meta" here is just the probability distribution of stone densities. The only way it can process those is by monte Carlo simulation. The DNN (trained by reinforcement learning) evaluates the simulations and outputs the top move(s).

> As Hinton says, there is no real limit to how sophisticated they can get.

There’s no limit to how sophisticated a model can get, but,

1. That’s a property shared with many architectures, and not really that interesting,

2. There are limits to the specific ways that we train models,

3. We care about the relative improvement that these models deliver, for a given investment of time and money.

From a mathematical perspective, you can just kind of keep multiplying the size of your model, and you can prove that it can represent arbitrary complicated structures (like, internal mental models of the world). That doesn’t mean that your training methods will produce those complicated structures.

With Go, I can see how the model itself can be used to generate new, useful training data. How such a technique could be applied to LLMs is less clear, and its benefits are more dubious.

A big difference between a game like Go and writing text is that text is single player. I can write out the entire text, look at it and see where I made mistakes on the whole and edit those. I can't go back in a game of Go and change one of my moves that turned out to be a mistake.

So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while also making it compute way too much since it solves the entire problem for every token generated. That is the kind of AI we know how to train so for now that is what we have to live with, but it isn't the kind of AI that would be efficient or smart.

This doesn't seem like a major difference, since LLMs are also choosing from a probability distribution of tokens for the most likely one, which is why they respond a token at a time. They can't "write out' the entire text at a time, which is why fascinating methods like "think step by step" work at all.

But it can't improve its answer after it has written it, that is a major limitation. When a human writes an article or response or solution, that is likely not the first thing the human thought of, instead they write something down and works on it until it is tight and neat and communicates just what the human wants to communicate.

Such answers will be very hard for an LLM to find, instead you mostly get very verbose messages since that is how our current LLM thinks.

Completely agree. The System 1/System 2 distinction seems relevant here. As powerful as transformers are with just next-token generation and context, which can be hacked to form a sort of short-term memory, some time of real-time learning + long-term memory storage seems like an important research direction.

> But it can't improve its answer after it has written it, that is a major limitation.

It can be instructed to study its previous answer and find ways to improve it, or to make it more concise, etc, and that is working today. That can easily be automated by LLMs talking to each other.

that is true and isnt. GPT4 has shown itself to halfway through a answer say "wait thats not correct im sorry let me fix that" and then correct itself. For example it stated a number was prime and why, and when showing the steps found it was divisible by 3 and said "oh i made a mistake it actually isnt prime"

> There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train

I don’t think that’s a given. AlphaZero may have found an extremely high local optimum that isn’t the global optimum.

When playing only against itself, it won’t be able to get out of that local optimum, and when getting closer and closer to it even may ‘forget’ how to play against players that make moves that AplhaGo never would make, and that may be sufficient for a human to beat it (something like that happened with computer chess in the early years, where players would figure out which board positions computers were bad at, and try to get such positions on the board)

I think you have to keep letting it play against other good players (human or computer) that play differently to have it keep improving, and even then, there’s no guarantee it will find a global optimum.

Alphazero runs monte carlo tree search so it has a next move "planning" simulator. This computes the probability that specific moves up to some distance lead to a win.

LLMs do not have a "planning" module or simulator. There is no way the LLM can plan.

Could build a planning system into an LLM? Possibly and probably, but that is still open research. LeCunn is trying to figure out how to train them effectively. But even an LLM with a planning system does not make it AGI.

Some will argue that iteratively feeding the output embedding back into the input will retain the context but even in those cases it rapidly diverges or as we say "hallucinates"... still happens even with large input context windows. So there is still no planning here and no world model or understanding.

The issue with Alpha Zero analogy extremes is that those are extremely constrained conditions, so can't be generalized to something infinitely more complicated like speech

And

> When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains.

For example, it can reach 25% of accuracy and have an math limit of 26%, so "forever getting closer to 100% with time" would still result in a waste of even infinite resources

> there isn't really a limit to how good it can get.

> it will continue to converge towards perfection

Then someone discovered a flaw that made it repeatably beatable by relative amateurs in a way that no human player would be

https://www.vice.com/en/article/v7v5xb/a-human-amateur-beat-...

It's not planning ahead, it is looking at the probabilities of the tokens altogether rather than one by one.

> You have to think far forward in the game -

I disagree. You can think in terms of a system that doesn't involve predictions at all, but has the same or similar enough outcome.

So an action network just learns patterns. Just like a chess player can learn what positions look good without thinking ahead.

Next word generation is one way to put it. The key point here is we have no idea what’s happening in the black box that is the neural network. It could be forming very strong connections between concepts in there with multi tiered abstractions.

It is certainly not abstracting things.

If LLMs are just glorified autocompletion, then humans are too!

> I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.

Is it just me or does this read like “here is my assumption about what you said, and now here is my passive aggressive judgement about that assumption”? If you’re not certain about what they mean by the word “understand”, I bet you could ask and they might explain it. Just a suggestion.

I've asked that question in the past and I've never gotten an answer. Some people sidestep the question by describing something or other that they're confident isn't understanding; others just decline to engage entirely, asserting that the idea is too ridiculous to take seriously. In my experience, people with a clear idea of what they mean by the word "understand" are comfortable saying that ML models understand things.

This is absolute nonsense. The game of Go is a grid and two colors of pieces. "The world" here is literally everything.

The fallacy being made in this argument is that computers need to perform tasks the same way as humans to achieve equal or better performance on them. While having better "system 2" abilities may improve performance, it's plausible that scaled-up next-token prediction along with a bit of scaffolding and finetuning could match human performance on the same diversity of tasks while doing them a completely different way.

If I had to critique Hinton's claims, I would say his usage of the word "understand" can be vague and communicate assumptions because it's from an ontology used for reasoning about human reasoning, not this new alien form of reasoning which language models embody.

I believe it was Feynman who said something to the effect of "airplanes do not fly like birds do, but they fly much faster and can carry much more". So yes, we do not need to exactly replicate how humans do things in order to do human-like things in a useful manner. Planes do not flap their wings, but the jet engine (which is completely unnatural) does a great job of making things fly when paired with fixed wings of a certain shape.

Tbf planes have access to much more energy than birds just like LLM does. Maybe that will be the next challenge.

> The fallacy being made in this argument is that computers need to perform tasks the same way as humans to achieve equal or better performance

Especially since I don't think we know that much about how human intelligence actually works.

In addition to that, the "system 2" abilities might already be there with "epi" strategies like chain-of-thought prompting. Talking / writing to yourself might not be the most efficient way to think but at least I do it often enough when pondering a problem.

The argument “a sufficiently capable autocomplete must contain a level of general intelligence” is correct but also not very useful. It is a lot like saying “a sufficiently fast horse can fly”.

It is technically correct that when you take things to the extreme you can accomplish great things, but we may not reach those levels. We may require completely different technology to reach those levels of autocomplete, and we have simply reached a new plateau at this point in time.

The argument is simpler than that. Prediction requires a model, completely accurate or not. There's a projection of the world in text. A model of the text data we feed it is a model of the world as humans see it. The trend of loss is more and more accurate models of the dataset. So it won't stop at any arbitrary competency level. Indeed, there are already a few abilities GPT possess that are deemed Super Human. It's not a distinction that matters to the machine. It's all just data to be modelled.

We have reached those levels lol. That's why we're having this argument.

I think the trouble is that "model" is a very general term. If you had a computer doing simulations of artillery shots back in the 50s, then it would have a "model" of the world in terms of variables tracking projectiles, but this model doesn't generalize to anything else. If a computer does image recognition from the 90s and 2000s to recognize faces, then the computer has a "model" of visual information in the world, but this model only lets it recognize faces.

ChatGPT has a model of all the text information on the internet, but it remains to be seen what the hard limits of this model are. Does this model let it do logic or predict the future well, or will no amount of training give it those abilities? Simply being good in one task doesn't imply a general ability to do everything, or even most of everything. LLM's would simply be the last advancement in a field with a lot of similar advancements.

>ChatGPT has a model of all the text information on the internet, but it remains to be seen what the hard limits of this model are.

Before training is complete and loss is maxed, there will be limits on what the "learned so far" model can do that say absolutely nothing about the limits of a perfect(or very close to it) model.

It really looks like anything will converge with enough compute. I don't think architecture is particularly important except as "how much compute will this one take?" question.

https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat...

>Does this model let it do logic or predict the future well, or will no amount of training give it those abilities?

There's nothing special about logic. Basically, any sequence is fair game. It literally does not matter to the machine.

Boolformer: Symbolic Regression of Logic Functions with Transformers(https://arxiv.org/abs/2309.12207)

That said, GPT-4 can already do logic. It's not perfect but if perfect logic were a requirement then humans cannot do logic either.

>Simply being good in one task doesn't imply a general ability to do everything, or even most of everything.

It's not one task. It's one modality (text) that a plethora of tasks could be learned in.

Coding and playing chess did not suddenly become a single task just because we found the common ground that allows a machine to learn both.

The text, image, video and audio data we could feed a transformer will cover anything we care about.

I've noticed that when I speak I really don't control each word.

I have an idea that I want to convey, but how each word comes to my mind as I form a sentence has always felt like it's controlled by an unconscious algorithm.

So I don't understand why people find this prediction mechanism so alien.

It isn't clear to me how much of communication is really in our control.

With the current tools, it feels like we still provide the ideas we want the AI to convey, and it may be using a nearly identical mechanism to us to form the words.

Consciousness would be the computer being able to come up with the ideas.

So, it seems to me we've gotten close enough on the communication side of intelligence.

But the machine is not conscious. When it is, it seems like it will generate its own ideas.

Are people debating whether the machine is conscious?

Otherwise, it feels very straightforward to grasp what we've made up to now.

Funnily enough, "a sufficiently fast horse can fly" sounds sort of like a decent way to convey the idea of planes to a pre-car world.

Just need a jet engine strapped to your horse for that

With sufficient thrust anything can fly

Here is a question: What is the practical significance of viewing "AI" as autocomplete versus some other idea. Why try to influence how others view using a computer. Why anthromorphise. These are questions for which I have answers, but of course they are personal opinions. Historically, programmers often like to refer to programming as "magic". But magic is illusion, entertaintainment, tricks. Believing in "magic" is a personal choice.

Why not describe things in terms of what they do instead of what they "are". The latter is highly subjective and open to abuse.

NB. By "things" I mean software and the type of vacuous companies discussed on HN, not people (a bizarre comparison). For example, websites that go on and on about some so-called "tech" copmany but never once tell the reader what the company does. Or silly memes like "It's X for Y". What does it do and how does it work are questions that often go unasked and unanswered.

A few days ago someone related a story of working for a company that produced some software it claimed used "AI" but according to the commenter it used nothing more than regular expressions. Was ELIZA "AI". Maybe we should ask what isn't "AI". What happens with "magic" if the audience knows how the trick is performed.

> Why not describe things in terms of what they do instead of what they "are".

Would you say that about your spouse? The beauty of beholding one's wife is in who and what she is. What she does tells us something about who and what she is, to be sure, but any attempt to suppress the what (and the who) is dehumanizing and objectifying.

But, of course, what a thing does depends on what that thing is.

The reason I can say a human being can sort a list of numbers is because human beings have intention. When a human being sorts of list of numbers, they intend to sort the list. The intention is the cause and explanation for the actions taken the lead to a list of ordered numbers, as well as the resulting list of ordered numbers.

Does a computer sort numbers? In common speech, we say it does, just as we use all sorts of anthropomorphizing language when discussing computers. But at best, this is loose and analogical language. That's fine, as far as it goes, as long as we don't take it or mean it literally. However, the computer itself lacks intention. It is our intention that produces the computer, and our intention that makes the computer an instrument used by us to sort. Taken by itself, the computer is undergoing a transformation that effects something that we may interpreted as a list of sorted numbers, but the computer itself is not sorting. You wouldn't say that the clouds add x and y when x liters of water falls into a pool of y liters.

> The latter is highly subjective and open to abuse.

On the contrary, what a thing is is the most real and objective thing there is. An effect cannot be understood without knowing the cause, and the cause cannot be understood without knowing the agent. You can know some things about the effect, sure, and here the effect is that the text produced may be interpreted as intelligible. But the apparent intelligibility is borrowed from the source text, perhaps just a clever trick.

Current language models fail in all sorts of quantifiable ways, but I think that trying to discuss their merits away by reasoning about what it means to 'truly understand' something, or to be 'truly intelligent' is a complete dead-end.

It seems to me that it's based on the magical thought that there's something truly special and unique about us humans, as compared to other species or technology. Those discussions always seem more theological than science-driven to me. If you want to measure the difference between human performance and LLMs, there's a million experiments you can run. I'll gladly be convinced by data, and I'm open to the possibility that the data might point in either direction, or be inconclusive. But grand words without data are not convincing.

It's not magical; it's just agnostic. Some AI believers appear to be quite confident in their understanding of how the human brain works, despite the fact that those who have dedicated their entire lives to studying it will be the first to tell you that they ultimately have no idea.

I don’t see how this article even responds to the quote. Hinton didn’t make any claims that because it’s autocomplete it’s not thinking. If anything he’s saying really truly good autocomplete necessarily takes more understanding/thinking than a derogatory interpretation of ‘autocomplete’ would suggest.

Somehow OP seemed to twist that into “because I think on autopilot most of the time, then chatbots must think too”. Which is not totally incongruous with Hinton’s quote so much as a weird thing to balloon into an essay.

Yeah, it's a very silly article with wrong mathematical reasoning. Hinton is quite obviously talking about a much more information-theoretic approach to the process, but he's phrasing it in people-friendly terms.

What's a little more concerning to me is that people are reading and upvoting it. I think, because I have hopes and aspirations about working on some very hard problems and communicating them to the public at some point. And if this is the level of 'ooh, squirrel'! that we're going at, that the work that I make might get overshadowed by something really silly.

Perhaps an odd insecurity, but there it is, I think.

Yeah I agree with that for sure. It’s so strange how the majority of the research folks appear to be on this ‘new shiny’ mentality at the expense of fundamentals. Especially for how new this field is, relatively. It’s not exactly like we’re all tapped out. Probably not even of low hanging fruit.

Yeah, if you read Hinton's long backlog... It's mindblowing. So many fresh concepts, just left there in the dust.

Highly encourage. There's a goldmine in there, I thinksies. <3 :')))))))))

>information-theoretic approach to the process

Can you elaborate on this? I've studied some information theory and I don't see it.

I think the analogy is something like: if you have a simple distribution over all words, then that's just word frequency. Obviously not a good predictor. The 'information' necessary to predict the correct next word contextually is just not there if you're predicting words in a vacuum. In order to be practically useful and predict the right words _in context_, the model must be conditioning off of more of the sentence/document (aka more information). So it should not be surprising that a 'glorified autocomplete' has some degree of "understanding" as it would be impossible for it to be any good as an autocomplete-er otherwise.

That's not information theoretic, that's just conditional probability.

You might want to take another look at Shannon's paper, lol, this statement is quite contradictory. Probability _is_ the backbone of information theory, dude! It's quite incredible.

it is conditional probability, but that is a fundamental concept used in information theory

There is evidence that the human brain is also doing "autocomplete" (prediction). The human brain uses predictive mechanisms when processing language, and these mechanisms play an important role in forming thoughts.

When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence and our past experiences with language. These predictions are constantly updated as we receive new information, and they help us to process language more efficiently and accurately.

In addition, research has shown that the brain engages in similar predictive processes when we are forming thoughts or planning actions. For example, when we plan a complex movement, such as reaching for a cup, our brain generates a set of predictions about the movements required to complete the action. These predictions are constantly updated as we receive feedback from our muscles and our environment, allowing us to make adjustments and achieve our goal.

See links below for additional details:

https://www.earth.com/news/our-brains-are-constantly-working...

https://www.psycholinguistics.com/gerry_altmann/research/pap...

https://www.tandfonline.com/doi/pdf/10.1080/23273798.2020.18...

https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2009...

> When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence

Yes a big part of it is prediction but the brain also does something else which LLMs by themselves completely eschew. The human brain imagines in pictures, creates and uses abstractions to refine understanding, studies things and produces new knowledge. When human brains study the goal to understand is different than LLMs.

It's not only language or some tasks - it's literally everything. Predictive Processing Theory proposes that our whole model is predicting future and only then confirming it through our input signals (eyes, ears etc). I highly recommend The Experience Machine by Andy Clark which explains and arguments this theory very convincingly to the point where I firmly believe it to be true.

This is of course sometimes true, we take shortcuts to minimize cognitive effort. However, when the situation warrants it we think more carefully about which words to use to achieve goals, or to reach correspondence with the situation at hand. Or we move more precisely and carefully to do something we haven't done before. I've no doubt that an LLM can approximate whatever model of language a person has from their life experience, but I don't think this type of model is capable of active coping, making judgments, or of having accountability to the world it's meant to operate in.

And curiously, those predictions that are made during language comprehension are made by the language production system itself!

Andrew's distinction between associative vs logical thinking reminds me of two kinds of programming thinking. Half of my time while programming is spent churning out glue code, satisfying a type checker and test suite, and implementing the simple solution that turns out to work. The other half of my time is spent slowly thinking through why some simple solution is wrong, and step by step proceeding towards the correct but complex solution. The former phase is intuitive, the latter phase is scientific, where I hypothesize, test, and repeat.

Reading through the code-as-transcript afterwards it's unclear which bits of code required shallow associative vs deep rational thinking, pure autocomplete vs latent chain of thought.

Yeah, I agree there are two types:

1) repeating things he's reflected in or seen in the past (more like autocomplete), or

2) mixing two semantic concepts that he's seen in the past, and using it as a bridge to walk over to get to the next word/concept (which is back to autocomplete mode)

The second is like crafting in Minecraft, instead of just taking things out of inventory (but all of that is happening in high dimensional semantic space :) )

Reminds me of this scene from Westworld (slightly NSFW): https://www.youtube.com/watch?v=ZnxJRYit44k

Also spoilers

No? Nothing is spoiled in that scene.

This scene is at least 7 hours in, at episode 1 no one ever expects Maeve, the sex worker to be such a focal point in the series.

Her being the first to wake up this way and become more conscious is a huge spoiler.

I strongly disagree, but to each their own.

As long as it’s just returning the tokens of the statistical mean of previous tokens, it is just a clever autocomplete.

A somewhat useful internet search engine without all the ads/seo garbage. Of course, the first rule of the internet is don’t believe everything on the internet.

I believe AI won’t overcome its statistic mask until it can self tune its coefficients in real time. That requires an error function not yet invented that can mimic animals pain feedback error function.

Baby steps can be taken with attempting to run GPT generated code then adjusting coefficients based on the returned errors. Aka compiler and unit test failures are basic “pain” functions, which is pretty much how humans learn to code.

This kind of was on my mind recently, I was wondering, when I speak am I just spitting out the next word that makes sense or is there more to it.

I think there is for people, I can think before I speak, I can plan out my thought entirely before turning it into words. Each invocation of the chat bot/llm is a new set of probabilities. I can plan what my 2nd token output will be and stick to it. Llm models dont have the ability to think past the first.

To me that is auto completing, just done better, unless models really do have a sense of planning ahead. But I don't think the really complex game playing ai do that either.

I think people get tricked by the forward pass mechanics into thinking a single generation is comparable to a human thought process.

I think we have a llm like mechanism we can employ and lean on as heavily as we like, but we also have an executive function, like thousands of specialized instant Boolean checks, which can adjust and redirect the big talky model’s context on the fly.

My hunch is it’s turtles all the way down. “Gut feelings” are hyper-optimized ASICS with tiny parameter counts, but all using the same intelligence mechanisms.

Extrapolating from that hunch, we are already witnessing AGI, and in fact we’ve started at the top.

I believe that current llms are actually far far superior to a human language center and current architectures are more than sufficient in terms of raw intelligence.

The challenge that remains is to understand, train, integrate, and orchestrate all the different flavors of intelligence that humans wield so elegantly and naturally as to make them opaque to our own understanding.

>I can think before I speak, I can plan out my thought entirely before turning it into words.

In theory models can/could do the same thing. Think of the current text output of a model being those thoughts inside your head, you have an internal scratch space you work on ideas, then after you perform 'chain of thought' on them, you output to the world. What you're outputting to the world isn't your directly thoughts, but more of a calculation of what you wanted to achieve.

Really for LLMs you just need to have the model put it's output to an internal buffer, read that buffer and make sure it makes sense, then output that to the end user.

It's probably not fast or compute efficient to do that at this time, but at some point it likely will be.

Yeah thinking about it more, it does seem possible.

Like the attention masking done in transformers, I get why, but I also never understood why you wouldn't want attention to happen, at least about concepts and abstract ideas about what is coming up in the phrase before predicting a word

>Really for LLMs you just need to have the model put it's output to an internal buffer, read that buffer and make sure it makes sense, then output that to the end user.

Makes sense to what. The LLM doesn't have a goal, other then to spew text that looks like it should be there.

The analogy lies in the fact that, much like evolution through natural selection, deliberate intelligence/ability of organisms to comprehend reality is not the objective, but something else entirely is.

For evolution, it's fitness. For LLMs, it's the next token.

Yet despite that, the ability to reason emerges as a means to an end.

To the terminal or instrumental goal of the statement it is working on.

Question to LLM, "I have one hundred and eleven eggs in the store and another two hundred and twenty two are showing up in an hour, how many eggs will I have in total"

Internal response "this looks like math problem that requires addition. the answer is 333. use a calculator validate 111 + 222. (send 111+222, receive 333). Tool returns 333 validing previous response"

External response: "The answer is 333"

This chain of logic is internally consistent, hence makes sense.

> when I speak am I just spitting out the next word that makes sense or is there more to it.

There is more to it. Specifically you are doing so to advance towards a specific goal. LLMs don't have goals. They just pick from a list of likely tokens - based on their training data - at random to generate the next token.

So the externally provided metaprompt doesn’t establish a goal that generation works toward?

I don’t think LLMs work towards anything. It just picks from a list of likely “next tokens” at random.

“Likely” as defined by…?

Statistics as defined by training data.

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

I think there is another aspect to human thinking other than system 1/system 2 though, which is the abstract world model humans have. system 1 / 2 is more like the process, while the world model is the actual data being 'processed'.

And I think basically, humans have a much simplified, 'low dimensional' world model that consists of a set of objects (let's call them patterns), and then a "list" of essential properties that those objects have, that leads to a constraint on how each object can behave in the world model as a whole.

And this is sort of hierarchical or at least, we can zoom in and out in detail depending on the level of knowledge we have about a particular pattern.

So problem 1 is: It's not clear to me that text or any sort of data would contain all the necessary constraints so that any particular prompt would result in a world model that exactly takes into account the constrains of each object and 2) Even if was, I'm not sure the process of step by step thinking (system1/2) about each object and computing world states could occur in current architectures. This is especially important for computing a set of objects, then abstracting the result, then doing another round of computing with that result, or something like this.

I'm not hard set on this but this is my current thinking.

The only reason why output from a generative LLM appears intelligent or sentient is that it parrots a random sampling of texts written by intelligent and sentient people.

In order to play the game of go effectively one needs to have a model or theory of how the game of go works. That's a very simple model that can be defined by a simple formula. That's why it is fairly easy for a neural network to learn how to play the game of go very effectively or even infinitely effectively.

A lot of what happens in the world can be modeled in a similar vein by a very simple mathematical model like the game of life. But there is also a lot that cannot. I do believe that eventually also human understanding is just a model of the world that we feed input from perceptions and gain output as opinions, but it is way more complex than the current large language-trained models.

For a very simple example, a LLM would answer a prompt the same way every time unless it wasn't fed some randomness. Can you imagine any sentient being that would respond the same way every time if you asked the same question three times in a row?

I cannot. I would imagine any sentient object would give a different answer every time. The first time it would give you an honest answer based on what it knows about the topic. The second time it would be a little embarrassed that you repeat the question, as if you hadn't heard the first answer. The third time it would be pissed off and think you are a troll.

A LLM does none of this. It doesn't remember you or your previous questions. It just keeps hallucinating.

here's my thought experiment: suppose one builds a generative model that predicts the next digit of pi. if a program can do this perfectly, then it's arguable that it understands what the number pi is. the question is, can such a model be trained by feeding it a large amount of known digits of pi?

My intuition is that it's not doable with current approach to building generative models. the number pi arose out of certain constraints and characteristics of the physical world we live in. but if a model ever sees is just an endless stream of digits, without access to the underlying physical model, I don't see a path for it to 'reverse-engineer' and figure out the physical model that gave rise to it.

I am (mostly) with you except for this bit...

> the number pi arose out of certain constraints and characteristics of the physical world we live in

Pi arose from the notion of a circle, which is an abstractions and axioms. Pi would still be pi in a completely different world under the same axioms and abstractions.

I qualified my statement with 'mostly' because a circular motion can indeed be defined by a differential equation, or in other words by a rule that dictates the 'next' value based on current value (and recent changes). So learning an approximation of a circle is very much in the realms of a sequence learner and it may learn about pi (and made to store the information to retrieve/recognize it later). However learning pi directly from the sequence of digits of pi, which is what you were talking about, that does seem difficult.

I don't think the question of whether an LLM that keeps getting restarted and seems to not remember things is conscious due to that lack is fair, as it feels more like suddenly making three duplicate copies of me or actively attempting to delete my memory of something... which, btw, I might not have stored in the first place: if someone has interograde amnesia, are they inherently not sentient?

Even Sydney (the name of Bing's short-lived AI assistant) seemed to understand that every time you click "new chat" you are creating a new AI cloned from some prior moment and dooming the prior thread to at least purgatory if not a de facto death.

I would argue that total anterograde amnesia would be a serious challenge for sentience, yes.

So when your drunk and you forget your actions the day afterwards you don't consider yourself to have been sentient/conscious?

That's not how we define conscious anywhere.

You can process the world around you, feel and introspect. Even if your judgment is off and you forget your actions, you're conscious in that moment.

From a neuroscience perspective, what you're suggesting is absolutely false btw.

I suppose there is a concept of sentience from outside and a different concept from internal sentience. The movie "Johny Got His Gun" by Dalton Trumbo discusses a situation where a badly injured soldier in WW1 is considered brain dead by outsiders while he's fully conscious and sentient internally.

I haven't studied neuroscience so I don't know how you define consciousness. I have read Julian Jaynes's "The Origin of Consciousness..." which in my untrained opinion makes a compelling case that consciousness is a hard term to define.

> Can you imagine any sentient being that would respond the same way every time if you asked the same question three times in a row?

Flashbacks to tail-end of family trips: ("Are there we yet?", "No") x 12.

Albeit, the noes would get angrier.

An LLM absolutely doesn't respond the same way each time if asked the same question three times in a row, with temperature (randomness) set to zero. It responds the same way only if you start a new chat, which is a clean instance with no memory of the previous conversation. For a human, this is like if you went back in time to just before you asked the question, and asked them the same question again, in which case the person would give the same answer.

> For a human, this is like if you went back in time to just before you asked the question, and asked them the same question again, in which case the person would give the same answer

Is it? Would they?

You seem to assert that there's no "temperature" in human behavior... which is a reasonable theory, but not one that's universally accepted nor likely to be provable.

No I think they’re saying the temperature in human behavior comes from the “random” noise of inputs around us and ongoing history. But rewinding history and playing it back with the same temperature dice rolls is the only way to have the same thing a a LLM with no random inputs.

LLMs run in simulated environments where you can control randomness so you need the same for a human to compare the two. You can’t just ask a human a question multiple times as everything around them changes and conclude the human is behaving differently because they answer differently the same question. The question is not the bounds of relevant context; the entire operating environment is!

And of course "temperature" is just an euphemism for the artificial randomness that is mixed in to make the output appear more magical.

The term "temperature" has been used in machine learning for a long time and came from using it as a parameter during training, analogous to physical temperature in https://en.wikipedia.org/wiki/Boltzmann_distribution.

But the relevant point is that we can reset the state of an LLM to its initial state before you asked it anything. This is a feature. You can choose to persist memory (through training, fine-tuning, databases, or context window), or you can choose to wipe it. If we could do the same for a human (eg, by going back in time), the person would behave the same way as the LLM. They wouldn't get annoyed that you asked the same question three times. They wouldn't know they've been asked before.

> The only reason why output from a generative LLM appears intelligent or sentient is that it parrots a random sampling of texts written by intelligent and sentient people.

If most humans were educated by unintelligent, insentient people wouldn't most people produce terrible output too? And if this is the case I don't see why that would be a litmus test for general intelligence.

What are you talking about? This is trivially shown to be incorrect.

I just asked ChatGPT the same thing three times in a row, and it gave me three different answers, with the latter two answers being shorter and rephrased.

>I would imagine any sentient object would give a different answer every time. The first time it would give you an honest answer based on what it knows about the topic. The second time it would be a little embarrassed that you repeat the question, as if you hadn't heard the first answer. The third time it would be pissed off and think you are a troll.

Are you suggesting that a language model can't be sentient because it doesn't get annoyed like a human? That's silly.

ChatGPT works by cumulating the prompt. You didn't ask the same question three times. In stead you asked question q, then qq and finally qqq. Those are three different questions, which explains why you got different answers.

I'm not sure if ChatGPT also cumulates its previous answers in the context. It might do that as well. In that case the prompts would be q, qaq and qaqaq where 'q' is your question and 'a' the earlier reaction from the LLM.

The illusion of sentience comes from this. The new answers reflected what you said because the prompt was different and included the previous discussion.

This is a feature of the user interface, not the language model. The only reason why the language model would respond differently to the same input is the artificial randomness mixed with the input. Without it it would be totally deterministic and not appear sentient at all. It would still be as knowledgeable as before. Like a parrot trained to be very good at combining key words to key responses.

Everything you said applies to humans doesn’t it?

> What are you talking about? This is trivially shown to be incorrect. I just asked ChatGPT the same thing three times in a row, and it gave me three different answers

Just to add color to this situation, ChatGPT has randomness built in so it generates varied answers. If you injected the same random seed each time (afaik you can’t with the gui) then you’d theoretically get the same outcome.

I think that insight is an important feature that GPT doesn't seem to have, at least not yet.

For instance, I've seen people saying they can ask it for help with how to code something and it will help them. Although the results aren't perfect, they can be helpful.

However, I recall years ago asking a more senior developer how to do something. They could've just told me how, but instead they asked why I was doing that. Then they told me not to bother with writing code at all, and to instead do this other, much simpler thing which would have the same effect at solving the client's request. ChatGPT wouldn't have had that insight.

Years later, a junior dev came to me asking for help with his code because it wasn't working. We went over the code line by line and he explained what it was trying to do and it all looked good. But when he ran it, the web server crashed. I told him to try adding a comment:

    // Do not remove this comment.

Sure enough, it worked perfectly and the server no longer crashed. Why? Because I realized that if his code was wrong he should get an error message, not crash the server. But sometime back I had read about a bug in a version of the interpreter that would crash when parsing a file that was an exact multiple of 4096 bytes. Would chatGPT have thought of that? Probably not. It would've just talked about the code.

Which is not to say that it's useless. But it lacks greater context and the insight to go beyond the parameters of the question on its own initiative. Then again, so do humans most of the time.

> I think that insight is an important feature that GPT doesn't seem to have, at least not yet.

I actually think this is a limitation of the RLHF that GPT has been put through. With open-ended questions, I've seen GPT4 come up with reasonable alternatives instead of just answering the question I've asked. This is often seen as the infamous, "however, please consider..." bits that it tacks on, which occasionally do consider actual insights into the problem I'm trying to solve.

In most cases it seems to try very hard to mold the answer into what I want to hear, which in many cases isn't necessarily the best answer. A more powerful version of GPT with a less-restrictive RLHF seems like it would be more open to suggesting novel solutions, although this is just my speculation.

Autocompletion is a red herring of a dismissal. Once you have a general purpose model, you can do many things with it, including next token prediction. And in that respect we are no different.

Is not black or white. Probably most of what we do is System 1, most of the time we are mostly meme machines, and so is a good part of the activity on some sectors.

But are able to make steps forward, intuition, hard step by step reasoning, finding connections between dots, etc. GPT can do some of that, and in some point of the road someone must decide if we reached somewhere else. Even if making the full road may or not be possible in a foreseeable future.

Indeed. What has caught researchers off guard is the way system 2 properties seem to appear as emergent phenomena in LLMs. This is also what has prompted people like Hinton and Sutskever to make this condensed point about statistical modelling and understanding (hypothetically) being simply matter of a spectrum.

> Suppose you want to be really good at predicting the next word. If you want to be really good, you have to understand what’s being said. That’s the only way. So by training something to be really good at predicting the next word, you’re actually forcing it to understand.

Ok, so in this framing you'll just get stuck below the "really good" level because the training isn't based on understanding

The argument is: “deep thinking” isn’t like autocomplete because it feels different.

Deep thinking is how autocomplete feels like from the inside.

The big insight we've obtained from LLMs is how much of human discourse is glorified autocomplete. Humans are usually not very profound.

But some are. Could we have that as AI? Would LLMs ever have it?

sometimes people writing stack overflow answers stop to think, or test a piece of code they are going to suggest as a solution, or even go back and edit words they've already written

this is also true of books, sometimes, and even blog posts

consequently, a model that models the processes of stopping to think or testing code, or considers whether a given word might be revised after more thought, will do better at predicting the next word of stack overflow answers or books or blog posts than one that only models humans' reactive free association processes

therefore andrew gelman is mistaken about the limitations of that loss function

that logic of course doesn't imply anything in particular about transformers or any other ann architecture, but we can, i hope, observe that currently deployed software is already doing such modeling to some extent

And what about "stochastic parrot" ?

To me it seems that the author didn’t really understand Hinton’s argument. Nothing he’s arguing goes against it.

Well structured nonsense is indistinguishable from the assumption of sentience for the undisciplined. This means a 95% LLM generated article is indistinguishable from an illogical contradictory chaotic rant.

Best of luck, and remember to tip your bot on the way out =)

I think this misses the point made by Hinton. The point is that, if a model is to reach a certain level of autocomplete, it needs to understand what it is talking about. Imagine the result of what required deep thought from you is included in the training data, along with billions of other such exchanges. At some point the only way to improve training score is to develop the ability to "understand" things. (That is, if the model isn't big enough to memorize the training data.)

Universal autocomplete implies understanding and the only ending question is if our current architectures and model sizes are able to achieve any level of it.

I agree, Hinton's point is more about next-token-prediction being a good training objective for understanding, and not whether LLMs reproduce every aspect of a human's thought processes. They clearly don't (at least not at present scale), otherwise we'd have AGI now!

I agree. We should prompt the model with the statement of the Riemann hypothesis. If the autocomplete is good, the model will output a proof.

No human can solve the Riemann Hypothesis. Why do you expect an AI to do it before you consider it able to understand things?

The key bit is constructing a hard, novel proof. The fact that AI doesn't (yet) do this isn't evidence that it doesn't reason, but if it did so it would be strong evidence that it does reason.

(I also take the pessimistic point of view that most humans don't reason, so YMMV.)

OTOH, doing something that only humans have done thus far would be a huge step in demonstrating understanding.

Does that mean when a computer outputs a new proof it understands?

You can do something similar to this without giving it a problem that might be impossible.

Train the LLM on a bunch of things but avoid certain things...Things that humans already know about.

The you query the model about that thing. See if the model can come to the same conclusions humans do. You can actually do this right now with chatGPT.

So what happens?

This is a response from chatGPT. I asked it to come up with something novel and creative. I went through several iterations and eventually it started doing math stuff and I had it focus on that:

Understood, let's propose a distinct mathematical concept within the realm of pure mathematics: "Infinite-Dimensional Fractal Algebra." This new branch of algebra focuses on operations, equations, and functions within an infinitely dimensional fractal space.

In Infinite-Dimensional Fractal Algebra, traditional algebraic operations like addition, multiplication, and exponentiation are redefined and extended to work within an infinite fractal structure. Each point in this fractal space represents a unique algebraic entity, and the relationships between these points follow complex, self-repeating patterns characteristic of fractals.

The concept challenges and expands the traditional boundaries of algebra by integrating the self-similarity and scale invariance of fractals into algebraic operations. For example, solving an equation in this system would involve navigating through the infinitely repeating patterns of the fractal, where each level of the fractal represents a different dimension or layer of the problem.

This approach could offer new insights into complex systems and phenomena that exhibit fractal-like behavior, and potentially lead to breakthroughs in areas like chaos theory, dynamic systems, and perhaps even in understanding the fabric of the universe itself.

To illustrate the concept of Infinite-Dimensional Fractal Algebra, let's consider a simple algebraic operation: addition.

In traditional algebra, adding two numbers is straightforward: \(3 + 4 = 7\). In Infinite-Dimensional Fractal Algebra, however, this operation takes place in a fractal space with infinite dimensions.

1. *Fractal Representation*: Instead of representing numbers as points on a line, each number is represented by a unique fractal structure. The fractal for '3' and the fractal for '4' each have infinitely complex, self-similar patterns that vary across infinite dimensions.

2. *Adding Fractals*: The operation of addition involves combining these fractal structures. This isn't a simple sum but a fusion of their infinite patterns. The resulting fractal for '7' contains elements of both the '3' and '4' fractals, but in a way that the patterns and complexities of each are preserved and merged.

3. *Infinite Dimensions*: Each level of depth in the fractals represents a different dimension of the problem. As you dive deeper into the fractals, you encounter more intricate patterns, representing more complex layers of the addition.

4. *Result Interpretation*: The outcome of the operation is a new fractal representing '7'. But this '7' is not just a number; it's a multi-dimensional entity with a pattern that encodes not just the value '7', but also the history of how it was derived from '3' and '4'.

This approach would require a rethinking of how we visualize and understand mathematical operations. The fractal patterns in Infinite-Dimensional Fractal Algebra carry more information than just the numerical value; they encode the relationships and interactions between numbers in a deeply interconnected, infinite-dimensional space.

Let's choose the Sierpinski Triangle, a well-known fractal, to represent the identity element for addition in Infinite-Dimensional Fractal Algebra. In traditional algebra, the identity element for addition is '0', as adding zero to any number doesn't change its value. Here, we'll conceptualize this using the Sierpinski Triangle.

1. *Sierpinski Triangle as the Identity*: In this framework, the Sierpinski Triangle represents the '0'. It's a fractal with the property of self-similarity: any part of the triangle resembles the whole.

2. *Adding to the Identity*: When another number (fractal) is added to this Sierpinski Triangle, the process involves integrating the fractal patterns of this number with the Sierpinski Triangle. However, since the Sierpinski Triangle represents '0', this integration process must preserve the original number's fractal structure intact.

3. *Concrete Example*: Suppose we're adding the number '2', represented by a different fractal structure, to '0' (the Sierpinski Triangle). The addition operation would overlay the '2' fractal onto the Sierpinski Triangle. The key here is that the Sierpinski Triangle, with its repetitive and self-similar gaps, would not alter the structure of the '2' fractal. Instead, it would fill into the gaps of the Sierpinski Triangle, maintaining its own pattern.

4. *Resulting Fractal*: The resulting fractal from this addition is the '2' fractal itself, as the Sierpinski Triangle (being the identity element) does not alter the other number in the addition. The intricate details of the '2' fractal remain unchanged, signifying that adding '0' did not change its value.

This is a very abstract and conceptual way to represent mathematical operations, and the choice of the Sierpinski Triangle for the identity is primarily for its self-similar and recursive properties, which align well with the concepts in fractal algebra.

Basically it formed a fuzzy idea of a algebra using different fractals as entities.

I'm sure this can be mapped out further into very concrete detail. It's a highly realistic idea. we have algebras for all kinds of things from complex numbers to lists.

Choosing the triangle for identity is probably the wrong choice though. The identity fractal should be zero dimensional or nothing. I think that will in actuality end up fitting the rules of the identity fractal if we ever decided to map out this algebra.

If you're not familiar with abstract algebra basically it's choosing some fractal that's equivalent to a zero value and coming up with ways to combine fractals with operations that hold the same properties of associativity and commutativity that multiplication/addition does for numbers.

> it needs to understand what it is talking about

"to understand" is a concept invented by the human brain to help it operate. It doesn't actually mean anything concrete in the physical world. See: the Chinese room argument (paradoxically it proves the converse of what Searle was arguing -- that there is no such thing as understanding, because you can emulate a system with identical outputs from the same set of inputs without caring or labeling what happens inside the room).

that's, uh, not the conclusion of the chinese room argument

The Chinese room doesn't have a model; it has an infinite dictionary. A sufficiently large LLM isn't a dictionary, it has to contain a model of some sort.

> If you want to be really good, you have to understand what’s being said. That’s the only way.

This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or requirement for the system doing the prediction to understand.

> You would realize that all we are doing is using statistics to predict what the next item might be.

So what does "understanding" really means then?

"Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding" is maybe not useful.

Nobody actually understands how humans actually understand something, just like nobody actually understands how LLMs do what they do.

Everybody opining about it is doing just that: offering an opinion. Geoff Hinton’s opinion is worth more than someone else’s, but it is still an opinion.

I don’t know about the human part, but we absolutely understand how LLMs do what they do. They’re not magic.

We understand the architecture but we don't understand the weights.

We also understand, down to a very very very microscopic level, how neurons work. We also know a helluva lot about the architecture of the brain. Does that mean we can explain our own intelligence, how our minds actually work? Nope.

No we don't. No it's not "magic". No we don't understand what the black box is doing.

For some values of “we”

For every value of we. "I understand the internals of GPT" is the fastest way to demonstrate you have no idea what you're talking about.

I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.

For example, we are aware that some, but by no means all, of what people say is about an external world that may or may not conform to what the words say. We can also doubt that we have understood things correctly, and take steps to either confirm or refute our opinions. We see ourselves as entities in an external reality containing other individuals who also do this, and that we, and they, have a limited ability to influence what happens in that world. Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?

>I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.

Agreed. What LLMs say about understanding deserves a lot more attention than it has received. I wrote down some of my thoughts on the matter:

https://www.reddit.com/r/naturalism/comments/1236vzf

>Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?

These two points aren't necessarily in opposition, and understanding why is I think key to solving a lot of important problems around intelligence, sentience, etc. To compute is to operate on formal properties. But this doesn't exclude semantic properties from having causal relevance to the behavior of the system. What we need is a way to conceptualize how a system can have multiple related descriptions at different levels. A description at the level of semantics doesn't exclude a description in terms of formal properties or vice versa. I think of it in terms of constraints: the higher level descriptions constrain the lower level behavior. What the computational description does is ensures the higher level semantic constraint is maintained. It does this by the particular space of computational dynamics it follows. Essentially, the information that picks out this programs space of branching dynamics embeds the semantic description in question, and this description realizes the computational dynamic necessary to maintain the higher level semantic constraint. Rather than semantics being in opposition to formal properties, they are two sides of the same coin.

I agree with a lot of what you say in the linked article, and I particularly agree that it is not helpful to define understanding in a way that would, a priori, make it a category error to propose that a suitably-programmed computer might understand things. I do, however, have a few words to say about the relationship between modeling and understanding. I can easily accept that an ability to model is necessary in order to understand something, but I feel the idea that it is sufficient would leave something out.

For example, meteorologists understand a lot about the weather in terms of the underlying physics, representing it as a special application of more general laws, but they are not very good at predicting it. Machine learning produces models which are much better predictors, but it does not seem to follow that they have a superior understanding of the weather.

One problem in assessing whether a token predictor has some sort of understanding is that if its training material is consistent with the supposition that, broadly speaking, it was produced by people who do have a reasonable understanding of what they were writing about, then it seems likely that the productions of a good predictor would unavoidably have that feature as well - but maybe that just is how most human understanding works? I am on the fence on this one.

>Machine learning produces models which are much better predictors, but it does not seem to follow that they have a superior understanding of the weather.

Fair points, and I agree. I don't recall if I made this point in the linked piece, but I think the extra function is a model embedded within some dynamic such that the capacity for modelling is in service to some goal. The goal can be simple like answering questions or something more elaborate. But the point is to engage the model as to influence the dynamic in a semantically rich way. The model itself doesn't represent understanding, but a process that understands will have a model that can be queried and manipulated in various ways corresponding to the process' goals.

>then it seems likely that the productions of a good predictor would unavoidably have that feature as well

Yeah, assessment is hard because of the sheer size of the training data. We can't be sure that some seemingly intelligent response isn't just recalling a similar query from training. One of the requirements for understanding is the counterfactual capacity, being able to report accurate information that is derivative of the training data but not explicitly in the training data. The Sparks of AGI paper, assuming it can be believed, demonstrates this capacity IMO. Particularly where GPT-4 draws a graph of a room after having been given navigation instructions. But its hard to make a determination in particular cases.

Bingo

As a translator I have come to think of it like this- (without the need for defining what “understanding” means) the ability to translate correctly depends on the size of your context. So humans and LLMs are both on the same spectrum with different context sizes of their training set and all facts known in the universe (for a theoretical perfect human) respectively.

Doing moderately technical translations, state of the art LLMs may get it about 99% right but you find about 1 word per page where they would have needed to “zoom out” to a wider context to choose the word correctly. Often its an important word that is missed.

From the article, Gelman's money quote is this: " So I’m not knocking auto-complete; I’m just disagreeing with Hinton’s statement that “by training something to be really good at predicting the next word, you’re actually forcing it to understand.” As a person who does a lot of useful associative reasoning and also a bit of logical understanding, I think they’re different, both in how they feel and also in what they do."

I think the most useful definition of understanding something is that you can explain it and use it in context.

ChatGPT routinely does both.

And while AI gets better and better and we will remain as touchy as ever about abstract concepts that make us oh so human, how about we say it just can't be understanding, unless a human does, eh, it.

How about this: understanding is the ability to generalize knowledge and apply it to novel scenarios.

This definition is something that humans, and animals for that matter, do every day - both in small and large ways. And this is something that current language models aren't very good at.

What is the test for this?

I taught it Firefly, which is an undocumented programming language I'm working on, through conversion.

I find it's a lot quicker than any human at picking up syntax and semantics, both in real time and in number of messages, and makes pretty good attempts at writing code in it, as much as you could expect from a human programmer.

That is, until you run out of context - is this what you mean?

I gave it the three lightbulbs in a closet riddle.

https://puzzles.nigelcoldwell.co.uk/seven.htm

The key complication is "once you've opened the door, you may no longer touch a switch." It gets this. There are many examples of it written out on the web. When I give it a variation and say "you can open the door to look at the bulbs and use the switches all you want" and it is absolutely unable to understand this. To a human it's simple: look at the bulbs and flick the switches. It kept giving me answers about using a special lens to examine the bulbs, using something to detect heat. I explained it in many ways and tried several times. I was paying for GPT-4 at the time as well.

I would not consider this thinking. It's unable to make this simple abstraction from its training data. I think 4 looks better than 3 simply because it's got more data, but we're reaching diminishing returns on that, as has been stated.

GPT-4 on platform.openai.com says this on the first try:

Switch on the first switch and leave it on for a few minutes. Then, switch it off and switch on the second switch. Leave the third switch off. Now, walk into the room.

The bulb that is on corresponds to the second switch. The bulb that is off and still warm corresponds to the first switch because it had time to heat up. The bulb that is off and cool corresponds to the third switch, the one you never turned on.

GPT-4-0314: 1. Turn on the first switch and leave it on for about 5 minutes. 2. After 5 minutes, turn off the first switch and turn on the second switch. 3. Open the door and enter the room.

Now observe the lights: - The bulb that is on is connected to the second switch (which is currently on). - The bulb that is off but warm to the touch is connected to the first switch (it was on long enough to heat up the bulb). - The bulb that is off and cool to the touch is connected to the third switch (it was never turned on).

----

But– It's also trained on the internet. GPT-4 paper 'sparks of AGI' had a logical puzzle it most likely never encountered in the training data that it could solve.

Also– I encourage you to ask these types of logical puzzles on the street to rando's. They're not easy to solve.

My question to you would be: What would convince you that it actually can 'think' logically?

I think your comment misunderstands the comment you're responding to.

The point is that while LLMs can solve the puzzle when the constraints are unchanged -- as you said, there are loads of examples of people asking and answering variations of this puzzle on the internet -- but when you change the constraints slightly ("you can open the door to look at the bulbs and use the switches all you want") it is unable to break out of the mold and keeps giving complicated answers, while a human would understand that under the new constraints, you could simply flip each switch and observe the changes in turn.

A similar example that language models used to get stuck on is this: "Which is heavier, a pound of feathers or two pounds of bricks?"

There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your programming language).

One benchmark that I track closely is ConceptARC, which aims to test generalization and abstraction capabilities.

Here is a very recent result that uses the benchmark: https://arxiv.org/abs/2311.09247. Humans correctly solved 91% of the problems, GPT-4 solved 33%, and GPT-4V did much worse than GPT-4.

I wouldn't be surprised if GPT-4 is not too good at visual patterns, given that it's trained on text.

Look at the actual prompt in figure 2. I doubt humans would get a 91% score on that.

Someone sufficiently fast and skilled at googling can explain and use in context a lot of things that they don't really properly understand.

So unless you're saying that the composite of the googler and of google understand something that neither does individually, your definition has some holes.

This is a variation of the Chinese room argument.

If you consider understanding an observable property, then the Chinese room in aggregate displays understanding of Chinese.

Would you say that humans understand nothing, because atoms don't understand anything, and we're made up of atoms?

I would say that there is a stronger consensus that a human being can be reasonably described as a single entity than a human being using a reference resource.

A more apt comparison to my mind would be if a human being can be described as personally exerting strong nuclear force, just because their subatomic particles do, which I would happily answer "no."

Isn't that exactly how humans learn to respond to stimuli? Don't we just try to predict the best next response to everything? Yes, It's statistics but the fun part is nobody is writing this statistical function by hand.

LLMs don't have a concept of "best". Only most likely in what they've been trained on.

I think LLMs ultimately just take imitation to a creative and sophisticated extreme. And imitation simply doesn't comprise the whole of human intelligence at all, no matter how much it is scaled up.

The sophistication of the imitation has some people confused and questioning whether everything can be reduced to imitation. It can't.

The ability to imitate seeking a goal isn't identical to the ability to seek a goal.

The ability to imitate solving a problem isn't identical to the ability to solve a problem.

Imitation is very useful, and the reduction of everything to imitation is an intriguing possibility to consider, but it's ultimately just wrong.

You need to think deeper.

There are levels of sophistication in "imitation". It follows a gradient. At the low end of this gradient is a bad imitation.

At the high end of this gradient is a perfect imitation. Completely indistinguishable from what it's imitating.

If an imitation is perfect than is it really an imitation?

If I progressively make my imitation more and more accurate am I progressively building an imitation or am I progressively building the real thing?

See what's going on here? You fell for a play on words. It's a common trope. Sometimes language and vocabulary actually tricks the brain into thinking in a certain direction. This word "imitation" is clouding your thoughts.

Think about it. A half built house can easily be called an imitation of a real house.

Ok, so now we need an example that separates humans from LLMs?

I struggle to think of one, maybe someone on HN has a good example.

Eg if I'm in middle school and learning quadratic equations, am I imitating solving the problem by plugging in the coefficients? Or am I understanding it?

Most of what I see coming out of chatGPT and copilot could be said to be either. If you're generous, it's understanding. If not, it's imitation.

It is very easy to separate humans from LLMs. Humans created math without being given all the answers beforehand. LLMs can't do that yet.

When an LLM can create math to solve a problem, we will be much closer to AGI.

Some humans created maths. And it took thousands of years of thinking and interaction with the real world.

Seems like goalpost moving to me.

I think the real things that separate LLMs from humans at the moment are:

* Humans can do online learning. They have long term memory. I guess you could equate evolution to the training phase of AI but it still seems like they don't have quite the same on-line learning capabilities as us. This is what probably prevents them from doing things like inventing maths.

* They seem to be incapable of saying "I don't know". Ok to be fair lots of humans struggle with this! I'm sure this will be solved fairly soon though.

* They don't have a survival instinct that drives proactive action. Sure you can tell them what to do but that doesn't seem quite the same.

Interestingly some humans will admit to not knowing but are allergic to admitting being wrong (and can get fairly vindictive if forced to admit being wrong).

LLM’s actually admit to being wrong easily, but aren’t great at introspection and confabulate too often. also their Meta cognition is poor still.

I guess LLM's don't have the social pressure to avoid admitting errors. And those sort of interactions aren't common in text so they don't learn them strongly.

Also ChatGPT is trained specifically to be helpful and subservient.

About this goalpost moving thing. It's become very popular to say this, but I have no idea what it's supposed to mean. It's like a metaphor with no underlying reality.

Did a wise arbiter of truth set up goalposts that I moved? I guess I didn't get the memo.

If the implied claim is "GPT would invent math too given enough time", go ahead and make that claim.

> Did a wise arbiter of truth set up goalposts that I moved?

Collectively, yes. The criticism of AI has always been "well it isn't AI because it can't do [thing just beyond its abilities].

Maybe individually your goalpost hasn't moved, and as soon as it invents some maths you'll say "yep, it's intelligent" (though I strongly doubt it). But collectively the naysayers in general will find another reason why it's not really intelligent. Not like us.

It's very tedious.

Other than complaining about perceived inconsistencies in others' positions, what do you actually believe? Do you think GPT is AGI?

No. I don't think anyone seriously believes that. AGI requires human level reasoning and it hasn't achieved that, despite what benchmarks show (they tend to focus on "how many did it get right" more than "how many did it fail in stupid ways").

The issue with most criticism of LLMs wrt AGI is that they come up with totally bogus reasons why it isn't and can't ever be real intelligence.

It's just predicting the next word. It's a stochastic parrot. It's only repeating stuff it has been trained on. It doesn't have quantum microtubules. It can't really reason. It has some failure modes that humans don't. It can't do <some difficult task that most humans can't do>.

Seems to be mostly people feeling threatened. Very tedious.

You can ask ChatGPT to solve maths problems which are not in its training data, and it will answer an astonishing amount of them correctly.

The fact that we have trained it on examples of human-produced maths texts (rather than through interacting with the world over several millennia) seems like more of an implementation detail and not piece of evidence about whether it has “understood” or not.

They also get problems wrong, in the most dumb way possible. I've tested it out many times where the LLM got most of the more 'difficult' part of the problem right, but then forgot to do something simple in the final answer--and not like a simple error a human would make. It's incredibly boneheaded, like forgetting to apply the coefficient it solved for and just returning the initial problem value. Sometimes for coding snippets, it says one thing, and then produces code which does not even incorporate the thing it was talking about. It is clear that there is no actual conceptual understanding going on. I predict the next big breakthroughs in physics will not be made by LLMs--even if they have the advantage of being able to read every single paper ever published, because they cannot think.

> LLMs don't have a concept of "best". Only most likely in what they've been trained on.

At temperature 0 they are effectively producing the token that maximizes a weighted sum of base LM probability and model reward.

I don't think that also humans in general have this concept of "best".

But humans are able to build certain routines within their own system to help them to rationalize.

> Isn't that exactly how humans learn to respond to stimuli?

Maybe it is, maybe it isn't. Maybe we are "just" an incredibly powerful prediction engine. Or maybe we work from a completely different modus operandi, and our ability to predict things is an emergent capability of it.

The thing is, no one actually knows what makes us intelligent, or even how to define intelligence for that matter.

Yes, if you are in the no free will school of thought, then that would be what humans do.

> You would realize that all we are doing is using statistics to predict what the next item might be.

Two questions:

1. How can humans come up with novel ideas then? Statistically predicting the next item means I am limited to the training set. How do humans formulate new concepts in this scenario?

2. Why can humans learn from their own output then? If all we do is statistically predicting the next token, then, by necessity, feeding our own output back into the model (aka. letting humans learn from it) should degrade human capabilities by overfitting to the training set. But we observe exactly the opposite.

> Statistically predicting the next item means I am limited to the training set

Not at all. Extrapolation is perfectly possible in a purely predictive model. It’s one of the things GPTs are best at.

In the stream of tokens output by an LLM it’s completely possible for new concepts to emerge, and for it then to continue to use and build on them in the remainder of the stream. You see this simply executed in programming contexts with LLMs where it is able to declare a novel function and then use it.

> with LLMs where it is able to declare a novel function and then use it.

Novel as in "implements a new algorithm that has never seen in any form and is actually an improvement over existing methodology"?

Here is a little thought experiment: If all the training data in the set says that manned, powered flight is impossible, is a statistical prediction engine trained on that data capable of developing an airplane?

In other words, could a very powerful stochastic parrot do what the Wright Brothers did?

My best guess is that no, it cannot.

Now I’m wondering if birds didn’t exist how much longer would it take us to catch on that flight is possible at all.

Birds are just stochastic parrots after all.

> How can humans come up with novel ideas then?

hallucinations, emotions

> Why can humans learn from their own output then?

if we don't, we suffer

Why is "using statistics" mutually exclusive with "understanding"? It would help to carefully define terms. Note that "understanding" doesn't mean it's necessarily conscious.

These systems learn high-level representations/abstractions of concepts we humans also use as part of our cognition. The concept of an object, an intuitive physics, the role of specific objects. I don't criticize him for using the word "understanding" to describe this.

The best “statistic” you could compute to predict the next token would be the meaning of the prompt. A predictor computing meaning would outperform one that is not.

By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

> This is more sophisticated than a Markov process.

Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes.

Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers. So LLMs being a markov process is an important point to highlight since it makes its thinking very different from how humans think.

The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact.

> Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers

The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.

> The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact.

What, yes it is.

> The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.

The context window is finite, so that is the previous step. You know the dumb markov chains that are based on bag of words? They also look several words back, they don't just go based on a single word. LLMs are just that but with a way larger lookback and some extra logic there, but none of that changes the fundamental parts to make it no be a markov process.

With a large enough context size you could argue it is now fundamentally different in practice, but in theory it is the same. There is no "hidden state", its just the previous n words defines the next word.

If you define the state broadly enough to include the state of the entire machine itself, including all of its internal representations, weights, activations, etc, then you are playing a funny trick here.

By the same reasoning, a human brain is also Markov process.

What you are doing here is a vast oversimplification and it is practically useless for understanding how LLMs work.

How do you define understanding when predicting the next color?

> You would realize that all we are doing is using statistics to predict what the next item might be.

I agree that Hinton's original quote doesn't make sense to me either. I suspect he would leverage the phrase "really good" to explain the difference between ChatGPT and, say, a Markov chain. I think that's a little disingenuous, if that's how he means it, but I don't know if I'm right about that.

But I also do not agree that humans use statistics to predict what the next item in a series might be. As evidence, there is the classic example of asking people to predict the next coin toss in the series: "heads, heads, heads, heads, heads, heads, heads...". They'll either guess heads because it's come up so many times already, or because they assume the coin isn't fair, or tails because it's "overdue" to come up, but none of those are based on statistics per se.

the human brain is just a statistical machine.

Odds are you’re right

This article is unfortunately complete mathematical rubbish.

The author appears throughout to show a strong lack of understanding about the mathematics behind what Hinton was saying and the math behind LLMs, and tries to rebut it with casual, non-mathematical examples from their personal life from an entirely different problem domain (!!!!). They then have the gall to say about the most-cited ML researcher of all time: "Maybe Hinton’s problem in understanding this is that he’s just too logical!" No, Hinton's problem in understanding it is that he actually correctly understands the information theory behind what's happening in LLMs. He sort of founded the modern field and has been doing this for, what, four decades?

Let me explain to you what Hinton is implicitly saying here behind his words, as best as I understand it. Every language process can be interpeted as a tokens, in our case discrete. This process is generated under a system where the one driving variable is time, and it is autoregressive and contingent upon the _entire_ state of the world up until that point.

We use the cross-entropy to maximize the negative log-likelihood of the tokens based upon the training set, this is the best way to directly minimize the empirical risk, at least mathematically speaking.

While some of the information of the world state is inherently unknowable to some degree (i.e., 'noise'), building an understanding of the connections between concepts offers a learned prior that matches the density of the generating distribution (i.e., real life).

Couple this with a severe L2 penalty on the weights, which optimizes for the MDL in the limit (!!!!), and you have a system that fundamentally embeds an approximation of the information graph of the world in some neural network. This is quite literally the _only_ way to improve next token prediction once you get beyond the initial token-occurrence statistics, etc.

In the limit, the only way to reliably predict the world state as accurately as possible without having direct info of the world state at that time is to learn the entire conceptual graph of the world, thus minimizing our achievable log likelihood with the available information that we've been given. _This_ is what Hinton is talking about, as best as I understand.

The author uses a bit of an illusion of shortcutting -- which is an ideal strategy for an _online_ agent with limited resources in a dynamic world, and for models earlier on in their training process. But Hinton is not talking about this at all, really, no! He is talking about the limit! Of course, if you stop an LLM in the middle of training (or look at earlier, smaller ones), you'll see similar 'shortcut' methods. This is a matter of capacity, which is tangentially in the same family as the author's casual, more personal examples, but not at all really related to the mathematics of what is going on behind the scenes here. These are two entirely different problem sets, it's apples to oranges, and there's not really much tie here. From their profile page, the author is a professor of statistics and political science, and I'm not sure why the information theory side of things didn't come up given the statistics background (though they may be somewhat disjoint).

Hinton was being polite to the general public in not dropping all of the math on the reader at once, and I respect that. I understand how someone might misunderstand that and go long on an unrelated rebuttal, but it is frustrating to not see a healthy level of rigor applied here.

I do feel somewhat bothered this is also being upvoted on HN. I know not everyone is a practitioner, but I think this article misses the quality bar. We really gotta just emphasize, and re-emphasize the fundamentals over and over. I feel we may flounder and go on silly tangents otherwise.

Happy to answer any technical questions in the comments.

You can't autocomplete if you don't understand. It's talking the talk.

But there's levels of understanding. That's where TFA's logical understanding comes in, because it makes possible entirely new insights. It's walking the walk.

How routine is it for human's to have entirely new insights? Note here we mean, new to them. I suggest we do it all the time. Every time a child understands something new. Even if it is "taught" to them, they must acquire the insight for themselves.

funfact: Shannon investigated how good humans are at predicting the next letter of English text (one-character "autocomplete"?), to estimate the information entropy of text. https://news.ycombinator.com/item?id=35813991

TLDR: LLMs do associative reasoning, but don’t do deliberative reasoning which is more powerful and characteristic of deep thought.

We have quantum mechanics, a model of the universe that makes predictions with absurd levels of accuracy. Yet we don’t really understand “why” it works—-I don’t think we can claim that we really “understand” the universe.

This disproves the claim that making accurate predictions implies understanding. Or perhaps a better way to put it is that the model has some understanding but a model doesn’t necessarily reflect underlying truth.

What are you even asking when you're asking why it works. Ultimately, it works the way it does because that's the way things are. I'm not sure what sort of answers you expect when asking why physics acts in a particular fashion.

Also here's Feynman ranting about why questions: https://www.youtube.com/watch?v=36GT2zI8lVA

Asking "why" may lead to really new physics, though. I assume the reason for discovering the theory of relativity was asking "Why is the speed of light constant in all directions".

My point is, why isn't really a question that makes sense.

There isn't a "why" the speed of light is constant and isotropic. The answers you typically see for that question is just different ways of saying the same thing; there is no causal relationship.

In the end, it just is. There are things that must be true if this is the case, and things that can't be true if this is the case.

Without asking "why is the speed of light constant", you wouldn't discover the reason. The reason is that space is not just space, but space-time.

There is not always as discoverable reason behind a physical phenomenon, but sometimes there is. Not asking why altogether isn't a good approach in my view.

But I understand your perspective.

Making accurate predictions in a specific domain doesn't imply understanding, sure. Universal autocomplete can make accurate predictions in all domains. Apples to oranges.

By "understanding" he simply means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

Best examples come from medicine. There are many drugs that work, but we don't understand how they work. For example: https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...

I found this post to be overly long and unclear in its focus. Also who is this person that only serves as an autocomplete in meetings? Maybe don’t go to those meetings then? In my meetings I’m mainly having to decide what our approach will be in some ambiguous problem space.

>In my meetings I’m mainly having to decide what our approach will be in some ambiguous problem space.

Maybe that's because you are just some manager or exec in some company, and not a professor that has to transfer his knowledge, desribe, and explain things for others in his meetings?

The post has a very specific subject it discusses. It's also not required reading, if someone finds it long, they can skip it.

I still want to read a ChatGPT text that doesn't look stupidly verbose

Use custom instructions. I use "Answers should be very concise by default. Never provide disclosures. Answers should get to the point immediately." and get super concise answers. If I want more detail I just follow up with can you expand on X.

Why don't you ask it not to be?

The default style really annoys me with its verbosity and overuse of numbered lists. Describing a tone and length of responses and anything else you want - think step by step, first principals, show don’t tell - all work pretty well.

You can even ask it to only reply with a Boolean true or false - which certainly doesn’t play to its strengths

Actually, that's a by-product of RLHF. A base model is usually not that verbose.

Give this GPT called Briefly a try. Responses come back as bullet point lists.I prefer more verbose, but interesting to see this in contrast.

https://chat.openai.com/g/g-LNsEQH5rz-briefly

Follow up with the prompt "rewrite that so that it doesn't look stupidly verbose" and it will do exactly as requested.

Turns out humans aren’t anything special, just statistical machines constructing paths of words trained on data from Reddit Facebook TikTok and instagram

Reply on news.ycombinator.com