Did that for old regional stories and the higher the score the easier I could verify the story or parts of it. It seems to know, at least in some cases
My impression was, while the resulting scores weren't entirely random, they were pretty much random when the input was difficult.
(I was mapping supermarket product descriptions to packaging materials - which is sometimes very easy, as sometimes it's obvious or the description includes explicit recycling information, and other times almost impossible, as for products like olive oil, glass and plastic bottles are equally plausible)
And if you ask gpt for an specific answer to something it can't actually know for sure it gets weird fast.
Asking it to guess how much of what it told is based on actual things it 'knows' at least seems like a reasonable question for a LLM with only one right answer.
Hallucinating should be about twisted input misinterpretation. E.g. if the text input you give them was malformed at some level.
What they are doing is try to give you an answer based on compressed knowledge, which might be true, but might not because it's compressed and lossy.
Because they are trained to bias to give an answer even if it's not 100% clear from compressed knowledge.
The larger the model, the less compressed the knowledge and better they would be able to differ when it is something they know vs something that they likely don't.
The former statement is a bold claim that would require some sort of evidence. The latter nobody really argues with.
Of course, it's not possible for an LLM to actually "know" anything in that sense of the word.
And if the probability is low it should consider that as hallucinating?
They can't detect if a word is high or low probability because current LLMs can't detect anything. They don't reason as you or I understand it. They are really fancy word predictors and somewhere in the word prediction machinery it does have the probabilities stored but not in anyway accessible to the output.
Imagine you are typing on your phone and the word autocomplete comes up and you just keep clicking the next word it suggests. What could you type to get at the inference data in your phone's keyboard?
Of course a system that does this could be built, but that isn't how current LLMs or phone keyboards are built.
When you ask an LLM to estimate if a given answer was hallucinated, it converts internal probabilities into a token(s) that represent these probabilities.
Here you have an output of chatGPT where it estimates it's probabilities of hallucination - which I would argue are quite close to correct:
https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb061700...
No modern LLM can query, let alone understand, the values in its own weights. They are simply inputs for the vector math. If they happen to be correct that is a either a coincidence or because it was trained with data including conversations about its past abilities.
Consider this, do you thinks the internal weights are labeled and directly to some attribute of the world outside? If so then present evidence. They are not, so if it does attempt to query its own internal weights how does it know what to query? This could be an interesting academic problem and I would encourage you to present papers on this because last I looked this was and itractable as of 2023 for the much simpler neural networks of simple backpropagation networks from 20 years ago.
Also, using ChatGPT as a reference here shows a major deficiency in your judgement. At best this is circular reasoning but likely you simply don't know how to allocate trust or how others do. The argument is X is unreliable then citing X is obvious not going to convince people who distrust X. Additionally, experts
That isn't to say that such an LLM or biological brain couldn't be built, but none presently are.
I'm as sure as I can be that she didn't know she was wrong; I can definitely tell you that I didn't know that I was.
1) LLM has data from a large set of sources, but in some cases it has very little data on certain sub-topics. With so few sources of data, it is easier for some weird corner of the internet to have conflicting data, and some LLM models may randomly favour that instead of the valid answer.
So not hallucination in this case, instead, a wrong answer from wrong sources.
2) LLM doesn't know, but has seen from human behaviour that just "making shit up, when you don't know" is entirely acceptable for many humans. Here I'm not referring to people saying (as I am) "may be", but instead "No, it's this!" with immense surety, in order to win discussions. Combine this with 'temperature' settings many model interfaces have, and there you go.
2b) Alternatively, #2 is inline with "I believe in flying spaghetti monsters", and other beliefs such as astrology, ghosts, and endless other "there is zero proof, but humans have immense weird data on it".
So #2 is solvable by changing temperature, or keeping temperature but saying "Don't make things up". #1 may be temp + random choice of a lower probability pathway.
#2b, not sure what to think here. How will LLMs ingest data, and use it to determine what is acceptable logic processing, when a lot of stuff in category 2b has zero proof behind it. And with zero proof, people are absolutely positive of the status of said thing?
#2b trains LLMs that "just having a random idea and then even killing people that don't agree, seems OK for humans". (think religious wars, violent acts, burning witches, etc)
LLMs do not truly learn or understand; instead, they use their internal data repository to produce "hallucinations" of text that seem conversational.
While LLMs do not truly comprehend or understand, their "hallucinations" can be remarkably useful and even enable them to accomplish complex tasks, such as solving mazes or translating languages.
In any case, I fail to see the relevance to my post. I only a) said that NNs can learn categories (an obvious fact) and b) conjectured that the categories "factual" and "creative" may be learnable.
[1] https://www.lesswrong.com/posts/WBdvyyHLdxZSAMmoz/taboo-your...
What if a psychologist said the same thing about a human? Should they be banned too?
LLMs can't ever understand anything anymore than a book can as they are static things. Until we move beyond transformers we're only ever collaborating in the hallucinations.
An LLM no mater how big can never have experienced any of this. The closest it can ever come is to have been trained on the tokens in texts relating to roses. If its more than an LLM it might have also been shown images or even video or roses but no LLM alone has ever seen a rose.
LLMs can only contain understanding the way a book can. There is no sentience in a book. Even if we jump to the old analogy of the Chinese room the instruction book in not conscious, nor is the reader following the instructions able to actually speak Chinese. It's beyond and between where lies the entity thar can converse in Chinese.
Because GPT-4o already can tell what is on images, so it is able to recognize a rose, and of course it has seen many images of roses.
It doesn't have the data for the odor, but it seems like arbitrary thing to train it with, for it to have a representation of the odor.
Would you consider GPT-4o to truly understand what a rose? And what if it was also trained on odor data?
But then again, this categorization provides no information to make any assessment of whether that is in any way achievable.
Dunning-Kruger effect
What makes you think that? The fact that you can ask them to stop hallucinating and that this does reduce hallucinations seems to me like a strong indication of the opposite? That is what the OP's article is about.
> They don’t know when they don’t know something.
The bigger models tend to be able to articulate some of their limits? See e.g this recent article [0].
> They know facts but they have no reasoning.
Is that really true? They can solve many novel tasks that seem to require some level of reasoning. They do not possess the full range of reasoning capabilities most humans have, but do pretty well on the HellaSwag dataset which aims to test reasoning [1], even when test set leakage was not a big problem yet [2].
I have 10+ years of experience in machine learning, and I don't find your claims obvious at all. In fact, I see more and more indications of the opposite. Also if you look mechanistically at how LLMs work inside, there is nothing stopping self-knowledge, reasoning or hallucination-detection. LLMs are very black box.
[0] https://arxiv.org/abs/2305.18153
[1] https://deepgram.com/learn/hellaswag-llm-benchmark-guide
[2] https://paperswithcode.com/sota/sentence-completion-on-hella...