Planning, factualness, logic - these are coincidences that arise because there exists an observer who can interpret the generated tokens.
Yes. Humans too can end up in a position where they are regurgitating words without understanding it. The converse, that because humans dont get it, therefore LLMs also understand, doesn't hold.
Edit: I too was deeply enamored and tried to create LLM minions for fun and profit.
However, it is when you move away from general data to production that the magic is stripped away. It’s prediction, not thinking.
The confusion arises because the emergent behaviors are considered human like and we fill in the gaps with human attributes.
We can see a big gap between the inner working of LLMs (which, unless I'm mistaken, are still not fully understood mechanically) and the inner workings of our brains. But many people assert that this is a qualitative difference, not a quantitative difference. (Probably this is true - we do things like math and spatial perception that seem independent of linguistic abilities. Meaning our brains are more like "LLMs, plus other types of AI mashed together with a central coordination layer")
However, nobody on Earth actually knows whether there is a qualitative or only quantitative gap between LLMs and human cognition.
Yes, I know how it works. I don't know why it works, though. Until LLMs I had a hard time understanding how humans work, too, but I think LLMs give us great insight into the mechanisms of intelligence. Are they the whole story of intelligence? Hopefully not. Do LLMs teach us anything about consciousness? Certainly that something can appear conscious without being conscious. It could be that I am the only conscious entity here, and you are all just LLMs.
If this token generation mechanism is put into a body of flesh, blood, electrons, neurons, will that be a conscious entity? I don't know. If the mechanism has a facility to reason about itself, along, let's say, 3 levels of nested awareness, will that make it conscious? I don't know.
I am sure you don't know either.
That isn't exactly how things work.
Certain things I don't know, and I've stated them pretty clearly above.
Understanding: The ability to comprehend or grasp the meaning of something, the capacity to apprehend general relations of particulars, or to interpret or judge information in a meaningful way based on prior knowledge, experiences, and cognitive processes.
LLMs (Large Language Models like ChatGPT) do not exhibit "understanding" in the same way humans do. They generate responses based on patterns in the data they've been trained on. They do not have consciousness, self-awareness, emotions, or experiences, so their "understanding" is limited to mimicking patterns without genuinely comprehending content.
I would agree.
I have extended your chat with ChatGPT, here is the transcript:
*system*: You are a helpful assistant. You can help me by answering my questions. You can also ask me questions.
--------------------
*user*: define "understanding" and explain if LLMs (including yourself) exhibit "understanding"
--------------------
*assistant*: "Understanding" is a psychological process related to an abstract or physical object, such as a person, situation, or message, whereby one is able to think about it and use concepts to deal adequately with that object. It involves comprehension, grasping the meaning of information, and being able to translate that into one's own words or use it in a practical way.
As for Language Models (LLMs) like me, we do not truly "understand" in the way humans do. We analyze input based on patterns and data we've been trained on, and generate responses based on that analysis. We don't comprehend the meaning of the words or concepts we process in the same way a human would. We don't have feelings, beliefs, desires, or consciousness. Our responses are the result of complex algorithms processing large amounts of data, not a result of understanding or comprehension.
--------------------
*user*: But would you say that LLMs still "understand" things, just unlike humans?
--------------------
*assistant*: In a very limited and technical sense, one could argue that LLMs "understand" things, but it's important to clarify what that means. When we say that LLMs "understand," we mean that they can process and analyze data in a way that allows them to generate relevant and often coherent responses. They can recognize patterns, make predictions based on those patterns, and use those predictions to generate output.
However, this is a very different kind of "understanding" than what humans experience. Human understanding involves consciousness, subjective experience, emotions, and a host of other factors that LLMs do not possess. LLMs don't have a sense of meaning or context beyond the data they've been trained on. They don't have personal experiences or emotions to inform their understanding. So while they can "understand" in a very narrow, technical sense, they lack the broader, richer understanding that characterizes human cognition.
--------------------
*user*: Of course, that makes sense. But if we agree that the world consists just of the data the LLM was exposed to, would you then agree that an LLM "understands"?
--------------------
*assistant*: If we were to consider the world as only the data an LLM has been exposed to, then within that limited context, one might argue that an LLM has a form of "understanding". This is because it can process and analyze that data, recognize patterns, and generate relevant responses.
However, even within this limited context, it's important to note that an LLM's "understanding" is fundamentally different from human understanding. An LLM doesn't have consciousness, subjective experiences, or emotions. It doesn't form beliefs or desires. It doesn't understand context or meaning in the same way a human does. Its "understanding" is purely a result of processing data, not a result of any kind of conscious comprehension or interpretation.
So, while an LLM can "understand" in a very narrow, technical sense within the confines of the data it's been trained on, it's still a far cry from the kind of understanding that humans possess.
====================================
I agree that an LLM is not conscious. I don't doubt that. If you require consciousness as a prerequisite for understanding, then an LLM does not understand. But I don't see a reason to do that, except to shield OpenAI from law suits.
In fact, I think that is a pretty good definition.
It's just like interacting with an LLM. You have to use things as "they understand them" - try to make up your own definitions to a word and see if an LLM can make sense of you. Of course they cannot (unless you do something like "this word now means this other word" which is pretty impressive, I have to say!)
The literal definition of "understanding" is "the capacity to apprehend general relations of particulars" (webster) which is expanded on by wikipedia "a cognitive process related to an abstract or physical object, such as a person, situation, or message whereby one is able to use concepts to model that object."
I would agree that a "model" is important here. I can model out code flow. A LLM doesn't seem to do this: it uses the code to predict more code (based on training data). This is painfully obvious in some cases, where no matter what you do, an LLM will start to "oscillate" between two wrong answers, or when you're trying to do something novel: LLMs can't come up with new concepts or new ways to apply existing concepts.
Now, talking about understanding a specific thing muddies the waters I think. There are plenty of things I do not "understand" or only "understand partially" but do I have the capability of general understanding? Of course. I argue that an LLM doesn't have understanding, but it can mimic understanding. In practice this may mean very little (LLMs certainly have more knowledge than I do on almost all subjects lol) but I think over time this will lead to a "wall" these systems will hit (if they haven't already).
It gets worse, that whenever someone comes up with a counterexample saying an LLM lacks understanding, the response is always "well, it just hasn't been trained enough, yet!"
Like, we agree - but for different reasons. I agree you can train it on that example (in the way that GPT4 seems to have been trained on chess games) and then it will start to look like it is "understanding" but I disagree it is capable of a intrinsic understanding. You can't teach LLMs a new grammar, for example.
I don't think we agree at all. I think LLMs do understand, you think they don't.
Most things are not well-defined, and if you want to be precise, you have to adapt your definition to the situation. That happens all the time even in mathematics. I think I gave you a pretty good one for understanding though: Let's assume we know what understanding means for humans, and then just use that knowledge to define understanding for an LLM. And if you use that definition, there is no way you can deny that LLMs do understand.
But even if we use your webster definition "the capacity to apprehend general relations of particulars", yes, LLMs do have that capacity. They are great at 1-shot, even 0-shot. That's exactly that.
2) Can you give me an example of you teaching an LLM a grammar?
for 2): no, I don't want to share that for now.
for 1):
*user*: What does "the capacity to apprehend general relations of particulars" mean?
--------------------
*assistant*: "The capacity to apprehend general relations of particulars" refers to the ability to understand or grasp the broader or overall connections or relationships between specific or individual elements or details. This involves being able to see patterns, trends, or commonalities among individual pieces of information, and to understand how they relate to each other in a larger context or framework.
--------------------
*user*: Do you think you have that capacity?
--------------------
*assistant*: As an artificial intelligence, I have the ability to analyze and identify patterns and relationships in the data I process. However, my understanding is based on programmed algorithms and does not involve personal experience or intuition like a human's understanding would. So, while I can identify and explain relationships in data, my capacity is different from human capacity to apprehend general relations of particulars.
--------------------
*user*: I don't see me asking if this is different from a human capacity or not. A simple yes or no answer is enough.
--------------------
*assistant*: Yes, as an AI, I have the capacity to apprehend general relations of particulars.
I am totally fine with you assuming whatever you want to assume.
But I wouldn't mind you giving me an example of how you failed to teach it a grammar!
I asked it to learn this basic grammar (then clarified when it made mistakes, and eventually got it to agree to a grammar like this): Let’s make a basic grammar. The things between brackets will be the set in our grammar. Our terminals are: [1, “, 2, X], all sets of strings end with the symbol: [Q], Our rules are: no terminals may repeat. When you have a terminal followed by another distinct terminal, the only thing that can come next is the repeat of those two terminals, and the string must immediately end. An example is: [1”1”Q]. There is no symbol for the start of strings.
After making a ton of mistakes and agreeing, it was very easy to trip it up. I said: "Great, let's only talk in this grammar from now on. When I give you a valid string, you give me the output. When the string is invalid, you say "Invalid" "
Then I did a bunch of invalid inputs (some of which it gave me garbage outputs), and eventually it got stuck saying "Invalid" to whatever I told it (just started to repeat that output, like LLMs do).
So, instead of giving it that description, give it a few examples of correct use of the grammar, see where it makes mistakes, and then add more examples / a description that would rule out these mistakes. It is really good at generalising from a few examples to the general case.
A person can be taught to understand this grammar - the machine lacks understanding.
I am not making excuses, I am explaining to you why it cannot do what you want it to do. No, it doesn't understand a grammar you give it in this form. But you can give it in a different form more suitable to its nature, and then it does understand.
But you don't seem to understand my point, yet I am not arguing you are not intelligent. You just don't understand certain things.
Build something with LLMs. Build something complex that depends on reasoning. I tried multiple times, they failed hilariously. I looked at others and those projects are also failing at exactly the same spots. The people who built OpenAI acknowledge that PoCs are easy to build but production is a huge challenge.
This is simply because the emergent properties make it seem like LLMs reason or plan.
Sadly I don’t have the energy to dig deep into the details - the crux of it is that humans bring semantic veracity - we are the observers that create a valid state through observation. LLMs are essentially Text based auto tune.
The fastest way to test this is to build something - like a team of independent agents. Even without getting into context window issues, you will VERY quickly see how semantically unaware LLMs are.
Im curious - How are you evaluating outputs?
There are basically two ways: a) You let the user check the output. If the user says, yay, this meets my needs, great. b) There might be an objective way to check if the output meets a certain objective. For example, you can let the LLM generate the output together with a proof that the output is correct. You can then check the proof separately.
And of course, you can mix and nest a) and b).
Sadly the Human Review option is simply never going to scale - Human review becomes the bottleneck. LLM output is non-deterministic at even temp 0, so scalable solutions are always critical.
Human review can work great depending on how you embed it into your workflow, and how qualified the user is to make a judgement. But yes, the goal is to reduce human review to the specification level.
Its not a dig at your knowledge.
ANd I can say this with some authority, because I have been trying to build LLM enabled tools that work only if LLMs can reason and plan. They don’t - they simply generate text.
You can test it out with building your own agent, or your own chained LLMs.
LLMs are analogus to an actors with memorized lines. They can sound convincingly like Doctors, but this is only skin deep.
To make it simpler - Karpathy said it in July, and the OpenAI CTO said it a few weeks ago - it’s easy to make PoCs but very hard to build production ready GenAI tools.
Our bags of flesh may be machines, however those machines are not simply biological LLMs.
If you try to measure if something is intelligent by trying to put it into a production ready workflow, then your measurement might be somewhat skewed... I mean, I am not trying to judge the intelligence of toddlers by putting them into a production ready workflow.
If a pig could converse with me at the level of ChatGPT 4, I am not sure if I could eat it.
> Our bags of flesh may be machines, however those machines are not simply biological LLMs.
I don't think we are just machines, and if we were, I don't know if we function the same as LLMs. But whatever it is that LLMs do, they are clearly intelligent, so they demonstrate one way that intelligence works. Harnessing this intelligence for something else than mere chats is a challenge, of course. I am using them as well to build something, and their current limitations are obvious. But even with these limitations, they allow me to do stuff I would not have thought possible a year ago.
I dont see how that is an argument.
You see intelligence, so I would urge you to build something that relies on that feature.
My philosophy was that the fastest way to figure out the limits of a tool are to push it. Limits describe the tool.
The data I have is on the limits of the tool. As a result its clear that there is no “intelligence”.
As I said before, is a baby intelligent? Of course. Could you use it for any kind of "production purpose"? I hope not. What about a 3-year old? You will have noticed that it can be difficult to get full-blown intelligent adults to do what you want them to do. This might even get more difficult with increasing intelligence.
And again, I ask you to put your money where your mouth is. If you are willing to assume I wasn't able to harness its intelligence, please prove me wrong.
There is nothing I would really want more, than to have genuinely autonomous systems.
My point at the start, and now is that by using Human terms to examine this phenomenon leads people to assume things about what is going on.
Testing and Evidence are what reason is built on. Asking someone to follow the scientific method, I would hope does not construe boorishness on my part.
The scientific method only works if you accept the evidence. Some people don't believe that we landed on the moon. Well.
You are telling me you could not use it for what you would have hoped to use it for, and you are not allowing the use of the term intelligent until an LLM can do that for you. If that is your definition of intelligence, good for you.
But I would suggest the following instead: What the scientific method has proven is that, if you feed a very simple mechanism with enough data, then intelligence emerges.
So what is thinking, then? Can you prove that humans aren’t just predicting what comes next given a series of inputs ?