This is the key thing with the limitations of LLMs at present, it can't "remember" what it has or hasn't done, by design.
If a math textbook has sentence x written in it, does the textbook know this information, or does it just have this information stored in it in a human-readable format?
LLMs only present their stored data in a human-readable format on the output, the data set itself isn't readable.
“It has no understanding”
I keep seeing this claim, I don’t understand what it’s supposed to mean. Do we humans have this magical “true understanding”? Can we test for it?
LLMs, as I (ahem) understand them, do not have models of the things they're writing about. They do have a model of language (or whatever symbolic system they were trained on), but that's where it ends.
I asked ChatGPT "Does an LLM understand, or is it simply Bayesian inference?", and the most relevant part of its output:
"The model's understanding of language is derived from the vast amount of text it has been trained on. By capturing statistical regularities and patterns, an LLM can generate text that appears to be comprehensible and contextually appropriate. However, it's important to note that LLMs do not possess true understanding or consciousness. They lack semantic or conceptual understanding in the way humans do.
Overall, an LLM employs a combination of statistical techniques, including Bayesian inference, and pattern recognition to generate text that simulates understanding to a certain extent."
When I asked ChatGPT "Does ChatGPT have a world model?", I got this output:
"ChatGPT, like other GPT models, does not possess an explicit world model. GPT (Generative Pre-trained Transformer) models, such as ChatGPT, are trained on a large corpus of text data from the internet and do not have direct access to real-time information about the world. They rely solely on the patterns and information they have learned from their training data.
While GPT models can generate contextually relevant and coherent responses based on their training, they do not possess up-to-date knowledge or awareness of current events. Their responses are primarily based on patterns, associations, and statistical regularities present in the text they were trained on, rather than an explicit representation of the world.
It's important to note that GPT models can occasionally generate plausible-sounding but inaccurate or misleading information, especially when prompted with questions or topics beyond the scope of their training data. Therefore, it's always advisable to verify information from reliable sources when seeking accurate and up-to-date information about the world."
When I asked it "Does a large language model infer an understanding of a topic?", it gave this output:
"A large language model, such as ChatGPT, can exhibit the appearance of understanding a topic due to its ability to generate coherent and contextually relevant responses. It can make inferences based on patterns and associations learned from its training data, which allows it to generate text that appears knowledgeable about a particular subject.
However, it's crucial to understand that this understanding is derived solely from statistical patterns present in the training data, and the model does not possess true comprehension or conceptual understanding like humans do. The model does not have access to real-world experiences, common sense, or the ability to reason in the same way as humans.
The responses generated by a language model should be interpreted as a combination of learned patterns and statistical associations rather than as genuine understanding. The model lacks the ability to deeply grasp the meaning, implications, or nuances of a topic beyond what it has learned from the text it was trained on. Therefore, it's essential to approach the responses generated by a language model critically and corroborate information from reliable sources when necessary."
The more balanced view is that we simply don't know, it's assumption on both sides of the fence—Schrödinger's world model. If something resembling a world model is emergent from the sheer scale of data, perhaps, that's a very interesting idea, but it's clear that it still doesn't understand, in the same way that a physics simulation doesn't understand what it's doing either, but it can still output useable information whilst doing something incredibly complex. Ergo, I'd say it's far more likely that there isn't anything out of the ordinary going on.
If I can't get something as simple as a valid NGINX configuration out of it without hallucinations, despite supplying it with the original Apache .htaccess file, documentation and the URL rewrites it will require, why would it have a world model? How an NGINX configuration works is much more simplistic than how the world works with its many systems. Especially when you consider that NGINX is inherently language-based, which should be what it excels at with pattern recognition et al, but it's dumb pattern recognition to a fault with "this commonly follows that" in the training data.
It has a model of the structure of writing on the internet. It can produce text that adheres to that model. That's all that's required for coherent writing.
That doesn't require or imply that the concept of "bar", "drink", or even "He" or the concept of motion has been modelled. The model is of the patterns of tokens (words/characters/whatever).
You and others continue to foist human qualities on LLMs because it looks like magic. To know is to know oneself. LLMs do not know themselves. In fact, there is no “self” to know.
LLMs are not self aware, I agree, but there is clearly some reasoning going on and you could argue that its model of human behaviour can somewhat simulate self awareness
What model of human behaviour? I asked ChatGPT if it has a model of human behaviour, and it said "ChatGPT does not have an explicit model of human behavior".
It just hallucinated an answer
If it didn’t have a model of human behaviour, then it wouldn’t work, because the whole idea is it’s simulating what someone acting as a helpful assistant would do.
Is every answer thus not a hallucination by this logic? If it's trained on external information, why would this information not include observations and information made about it? A human doesn't need to be able to even read to present a book to someone else with contextually relevant information.
But this is true of humans in many cases too
It’s training data is from 2021, so it won’t contain anything about how ChatGPT works, maybe a bit about how LLMs in general work though
To wit: I'm cofounder and senior researcher at The Center For Decentralization. We have business cards, phone numbers, offered a free consultation service to budding decentralists who were interested in shaping public policy, a social media presence, the whole deal. For a little while we even sent representatives to Congressional meetings and had an office in DC. I have no official credentials in this area. And I'm also the only co-founder, because a decentralist mustbtake care to avoid any language that makes it sound like you're the center of the thing you're trying to decentralize. Which, naturally and inescapably, you are.
I started TCFD as a parody to point out the fundamental paradoxes arising when one affects to be sponsoring or causing decentralization to happen, whether as an individual or a group of individuals. I thought the joke would be obvious to anyone the moment they saw our business card or heard the name, but was rather astounded to to discover that about half of the people I introduced myself to as "senior researcher, primus inter pares" completely took it seriously.
I do not, and based on the direction of ML progress from the last couple of years I am becoming more and more certain that it will have a low impact on AI, if/when that materializes.
Therefore I won't scoff at someone that's not involved in ML but still considers themselves as being part of the AI world, at least not without more evidence that they're an impostor. Take John Carmack for example, at which people look down in similar ways, because has no formal training in ML.
I think AI field at present is in an pre-Enlightenment period, where people studying astrology, alchemy mix up freely with astronomers, chemists and philosophers and it will take some time until we can discern the useful from the useless, or from the outright harmful.
https://www.npr.org/sections/health-shots/2022/10/14/1128875...
This has my attention, particularly from an ethics perspective in terms of slavery.
> "No, I did not write that poem."
Literally what the comment is saying. It wrote something and can't remember that it wrote something. Remembering that it wrote something is fundamental to answering the question of if it wrote something. Otherwise, how would it ascertain the difference between someone whose style mimics that of ChatGPT output and actual ChatGPT output? No memory means no certainty.
It's different from asking it if a comment was generated by an LLM, in which the response heavily leaned on the word "appears". However, LLMs have no understanding of what it's given, it's all just tokens and Bayesian inference.