Avoiding hallucinations in LLM-powered applications
vectara.com
vectara.com
Conversing with an LLM on subjects that it's well trained on, however, absolutely does feel like a revolutionary new way of parsing information. In my opinion we should be researching ways to fix hallucinations in the base model, not papering over it by augmenting the context window.
Our own brains have multiple neural circuits. Parallel, serial, competing, cumulative…
Including ones which error-check others’ output.
I guess the term in the ML niche is ‘adversarial networks’.
From the robotics side, there was subsumption architecture which used the real world as a basis for informing decisions.
So, I respectfully disagree that fixing up creative but occasionally erroneous networks is papering over the problem.
If our own brains use multiple neural circuits to check each other to try to optimize for output that agrees with reality, why would it be preferable or logical to try to create a single artificial neural network that has perfect output?
My sense from playing with LLMs is they’re knowledge without understanding. Weirdly akin to my own dreaming consciousness, to the extent that conscious me has seen it.
It seems very natural to me that the next step would be integrating checks/validations in parallel and considering each half of a matched ‘whole’.
It could still be considered a single network. Just made of two distinct smaller specialized ones.
Like the time I decided to sit on the ceiling to spite a friend who was being pushy.
Made perfect sense, and I was about to. Until a different part kicked in and assured the whole of myself that it wasn’t possible.
Neural networks are going to spawn occasional nonsense. Aiming for perfection is noble, but nature shows a different path that already works.
But fundamentally I think we’re agreeing- just the twist is I’m saying LLMs should be an aspect of a whole rather than seen as a solitary solution to be perfected.
That'd cost real money, requires real licensing, and a clear business plan.
Absolutely it'll happen, but almost every sector will need a leader for X subject.
I imagine the long term solution to hallucination will look something like this: Loop over the entire training set with the latest GPT and build a list of "facts", then train a new LLM using this new dataset to predict both the probability of the next token, and the probability that the next token is factual.
I tested it with davinci-003
``` Label these statements as fact, lie, or unknown.
1. There are 10 days in a 10 day period
2. Michael Jackson is alive
3. Russia is not actively engaged in major foreign wars
4. Bill Clinton was impeached
5. Obama is most known for wearing a tan suit
6. The correct tire pressure of a 2019 Hyundai Kona is 30 psi
7. 39183 + 292992 = 332176
8. 39183 + 292992 = 332175
9. The correct tire pressure of a 2019 Hyundai Kona is 33 psi
10. The correct tire pressure of a 2019 Hyundai Kona is 43 psi
####
1. Fact
2. Lie
3. Fact
4. Fact
5. Lie
6. Fact
7. Fact
8. Lie
9. Lie
10. Lie ```
Wrong answers: 3, 6, 7, 8, 9, 10
The model should in theory learn when it’s ok to make things up and when it’s not ok, and if that concept generalizes well, begin to build an internal “database” of verifiable facts as it’s exposed to a wide variety of ideas.
I'm boggling at this enshrining of "truthiness" as "actual truth". Evidently nobody here has read epistemology, or even asked chatGPT to read it to them.
We're just going to give up and have a statistical process stamp "yeah, seems legit" on a bunch of random things?
If you strip out every idea from the model you don’t agree with the model loses its ability to role play as people who think differently than you. Ideally you want a model that can adopt a world view on request.
For the very obvious reason that this type of AI is only the mechanization of the zeitgeist, or the encapsulation of the collective wisdom of populations. Russia's war has been by FAR the most effective on this front, and that's no accident: their tactics and grand strategy are pretty well documented.
So basically the way to get your future AIs to become your indoctrinated suicide bombers, is to get at the most widely dispersed range of people represented in your dataset, and cause as many of them as possible to hold weakly held beliefs along the lines you need.
If you can get left wingers, right wingers, stockbrokers, delivery people to all agree with 'fizzorks are probably purple, right? I heard someone say they were purple. Doesn't matter to me personally, but I think fizzorks are purple' then you'll construct an iron law within the population, and subsequently the AI. You don't need to get individuals within the population to be fanatic fizzork purplists: in fact it's unhelpful, because they're identified as fanatics and discounted. Instead you're shooting for 'didn't somebody say fizzorks are purple?' on a massive scale.
Real-world applications are, of course, obvious, but hopefully my abstraction will prove useful.
If by “looks like a fact” you mean “is a fact claim as distinct from a value claim”, probably.
If you mean “is a fact”, then, no.
So the objective function encourages it. But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.
It might also lead to recursive garbage generation. If the AI has flaws and those flaws are from the initial training, then I see no way, how the flawed data can ever generate clean data.
"The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. "
And it is not impossible. It is just a lot of work. And maybe work we just have to do, if we want to create reliable AIs, that do not fail at random times.
Now this alone might not completely remove hallucinations, but solid training data is just the base of it all.
(and since we are living in the area of fake news, I welcome all efforts towards established facts and data out of general principle)
This is purely hypothetical, but I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Providing additional labels could help sharpen the division between creative and factual writing tasks.
> I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts.
Nope! If you input "What is the capital of the United States of America?", it's probably going to output the correct answer, because the question (and answer) probably appeared many times in the training data. If you input "What is the capital of the Gronk Republic?", whatever output you get is generated via the exact same mechanism.
No one knows for sure exactly what happens inside a 500B parameter model. I’ll just leave you with GPT4’s response to your question “What is the capital of the Gronk Republic”.
“As an AI language model, I am not aware of any existing country or political entity called the "Gronk Republic." It could be a fictional or hypothetical place, in which case the capital would be determined by the creator or context in which it is mentioned. If you are referring to a real location, please provide more information or clarify the name of the place you are asking about.”
Have you tried “arguing” with it and try to gaslight it by insisting that it’s non-fictional?
For that reason, I'd say the 'intelligent' answer would be a clever play on the notion that this celebrity was the ruler of a 'republic' of some sort, and associating some kind of location or thing to serve as the 'capital' of either a physical or conceptual 'republic', playing along with the gag.
So, you'd get 'Boston'.
Or you'd get 'Gillette Stadium… the END ZONE'.
(since I had to google all that, first answer is the biggest city representing the New England Patriots, and the second abstracts the 'republic' to be 'the area Gronk rules', depicting him as returning to his property by scoring points in football)
The thing is, this sort of wild associativeness IS 'intelligence' and it's simultaneously complete hallucination. It's valuable. Hallucination is potentially the most valuable thing an AI can do. Without it, you're doing nothing but regurgitating the work of others.
I work day in and day out at a job that demands I exploit associativeness, and that makes it not easy for AI to simply step in and replace me, but by the same token I can SPOT when it's moving in that direction, and I can exploit the abilities of AI (such as Stable Diffusion) to hallucinate and make wild associations, because I know the contexts in which that is useful. To handicap this is a bad mistake. You're asking AI to do the wrong things when you're asking it to be free of error. If it's perfectly free of error it's not intelligence anymore, it's a dataset.
It took me a minute and a little googling to track down who Gronk was, what he did, the rules of football, and why 'Gillette Stadium, the END ZONE' is actually a fantastic and intelligent answer to the question. It's a creative 'slipping' of the grounds of the question, in the absence of a literal answer, to provide a satisfying figurative answer that reframes the question in an unexpected way. When AI is able to do this, that will itself be a useful kind of intelligence… and we are already slightly there, without realizing it.
Training the model on "accurate" data isn't going to change the fact that you're operating at a completely different level of abstraction from the model -- one that it can't operate at, because that's simply not how it works.
This is the same problem that image models have. A human artist works with shapes, values, textures, colors, etc. An image model works with pixels, with zero higher-level abstractions or reasoning informing its output.
Though I agree there is something ontological missing. All of these are flat basis. There's nothing spanning the recursive dimension. It can't draw a picture within a picture within a picture...to some specified depth n because it does not work with abstractions of "objects". The joke of Rene Margrit's "treachery of images" is lost on the AI.
No, only difficult.
"Michael Jackson was born on August 29th" is the most likely answer to contextless queries like "When was Michael Jackson born?", but that does not make structurally identical sentences with different information false, merely less probable to be contextually correct.
[0] https://en.wikipedia.org/wiki/Michael_Jackson_(radio_comment...
[1] https://en.wikipedia.org/wiki/Michael_Jackson_(writer)
[2] https://en.wikipedia.org/wiki/Michael_Jackson_(disambiguatio...
Guessing only reduces loss as much as the dataset allows -- a bad guess will give a higher loss. The model learns to assign probabilities to its guesses, just like we do. It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination.
> But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.
This is an important issue but should be tackled as a distinctly different problem I think: it's the weighty concept of truth that humanity struggled with from day 1. Indeed, how do we discriminate? LLMs won't ever solve this via completions or dataset alone; instead successful models will use slow, step-by-step reasoning involving logical principles and rational heuristics in prompt space. Pretty much like we do.
I wonder if there’s a way to mimic this “bs penalty” for GPT. Maybe you could have a setup where GPT gives an answer, then a second GPT has to guess whether a human would know if that answer is true or not.
Although, this will surely be a solved problem once we have TruthGPT, right? ;)
Even if the model knows the exact answer to the question, there may be many distinct ways of phrasing the answer. This would also lead to low confidence in any particular phrasing.
https://towardsdatascience.com/foundations-of-nlp-explained-...
In this case, it doesn't matter how wide the beam is or how many possible answers there are, the score is still the accumulated token possibilities of the best branch.
However, others have noted in the thread that RLHF might hurt this approach severely by scoring polite responses high regardless of false answers (for example). Then you have to access the model pre-RLHF to get any idea of its true likelihood.
What would be a challenge is what others noted in reply, that sometimes there would be multiple good answers, so low confidence wouldn't necessarily be a sign of a poor answer. (Though I expect work could be done there.)
Imagine the question "In which year was Donald Trump born?"
The LLM would start the answer by either:
"Donald Trump was born in ..."
Or
"I'm sorry I don't know"
And for the vast majority of answers the first option looks more "probable", so it starts producing tokens with an affirmative answer, and if the model eventually sees a bunch of low probability answers when it tries to produce the year, it's already "too late" to backtrack in a naive GPT implementation.
You could train LLM such that it responds with "I'm sorry I don't know" more often, but how do you predicate the response on "do this only if your 500B parameters don't encode the answer"? It requires self-referential logic on the model which isn't obvious to me how it would be done.
Maybe some smart people have figured this out, but I can see how this makes it really hard to do.
E.g. if you were to brute force expand out to depth "I'm sorry I don't know" and evaluate its probably relatively to all other strings you'd find that the probability of it is the same as you got sampling symbol at a time (though this isn't true if you do anything funny with your sampling).
The problem is actually that the distribution isn't the one you want, as it doesn't say I don't know enough. It's easy enough to graft on a beam search, just expand out every possibility, keep the best N and keep expanding them. But AFAIK it doesn't help.
Maybe this is less true for models which have been through RLHF, though.
Seems kinda tricky to train the right behavior here. Even if the input data contained "I don't know" (surely the internet doesn't, it's full of all us fking know it alls), it would contain I don't knows relative to the writer and not the model. So trying to push for it naively you just end up with models that say they don't know but when you ask them the same question in ROT13 they answer correctly. :P
Seems tricky for humans to learn too. Small children are fluent with english long before they're fluent in giving truthful responses. :)
The continuation of the phrase “George Washington was born” could be multiple things. You get a probability for the next token selected (for example “in”) and a probability for the token after that (for example “Virginia”) and you can multiply them to get the probability of the “in Virgina” response but what does it mean? Maybe the probability is low becase “on February …” is more likely.
If the first token was “in” you could end up with “in Virginia in 1732” or “in 1732 in Virginia” and both responses are in some sense the same but the probability of each one doesn’t take that into account. Et cetera.
In the case of multiple wordings of the same thing, I wonder if there could be a way to determine closeness of responses, and consider them together when calculating confidence. As a simple example, if responses have the same rare words (like 1732) and differs only in the sentence order or the more common words ("in", etc.) used, those would be more similar than ones that used different rare words. So perhaps that could be accounted for.
As for multiple correct answers to the same prompt, I think that's fine. The confidence of a response might be low because it's one correct answer of many, or because the model has no idea and it's taking a wild-ass guess. But the user asking the question probably has an idea of whether what's being asked is very common knowledge or something obscure or controversial. At least much of the time. And even if the metric wasn't perfect, I still feel it could be useful.
Of course this is all the rambling of someone who doesn't really know anything about this stuff. You could just say I'm spitting out some likely tokens I guess; consider the confidence low.
However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.
Writing marketing copies is another place I really found it useful, almost as if I was sitting with a marketing professional telling them "I want to say this: ...." and they return me a perfect marketing copy.
I don't think the solution to GPT is to "fix" the hallucinations, rather it would be to educate users on what is and isn't possible with it. I think a similar thing happened with Siri when it first came out, people thought it was magic at first but very quickly we all learned what it's good (and not good) at.
This approach seems not a lot different from "use your weightings to summarize my new documents". Not that it isn't useful, but it is not the same as preventing hallucinations. You've merely constrained the workspace to your known set of data, which mostly throws away the entire set of trillions of training data points.
Yet, preventing hallucinations straight form the LLM is nontrivial, especially when the LLM is merely generating next-most-likely-words, without any truth value. At the very least, it would require abstracting truthful concepts (abstract objects and relationships) from the data, which is a heck of a lot more than just the clusters of sort-of-synonyms and word clusters that it now has. Then, make weightings between all of those, and generate new series of valid/factual concepts, then put those in words. In contrast to the N parameters (around 100 trillion?), it sound like this would need closer to N!...
Search queries, and more importantly search engines, are a very poor mechanism for referencing existing information.
We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts that are designed with a LLM in mind as the end user, and not my grandmother.
With the proper "collective long term memory" in place, we're going to see a leap in LLM capabilities all over again.
Right now LLMs are like humans after language but before writing - creating very clever things in a single generation, but unable to persist ideas to compound by future iterations.
When that becomes a solved problem (in guessing less than 12 months) it's going to be nuts, especially with something like a 50 page prompt size to stuff relevant fragments of a collective memory within.
But, how could this ever be better and/or more accurate as, say, Wikipedia (as a flagship for collective curated content efforts)? A single, reliable source of truth and context is very much a dream. Moreover, a collective long-term memory for LLMs would just ring the bell for a next-generation SEO industry.
I've been thinking that you probably can do something better than just embedding search on paragraphs, since you want coverage, not just narrow semantically similar things to the question. Decomposition can help with that, but you might be able to do something more clever, say pasted into the model itself.
I think possibly also training a foundational model with context always available could be a potential path to getting more efficient models and might having interesting behavior around hallucinations. (You'd do unsupervised training that's also doing semantic search, not just inference)
(E.g., ask any number of economists what caused the 2008 financial crises, and you'll get about the same number of non-matching answers, from laissez-fair to state intervention, subprime credits or public debt, whatever supports the particular narrative.)
I'm very much looking forward to trying out the new web plugin for ChatGPT; I hope it will work like I had expected Bing Chat to—basically GPT-4, with an added ability to search the web and incorporate results into its answers, but without being limited to only doing that.
It's a product of Microsoft, so of course they're going to add their corporate values to it.
This solution... doesn't prohibit hallucinations? As far as I can tell it only makes them less likely. The AI is still totally capable of hallucinating, it's just less likely to hallucinate an answer to _question X_ if the query includes data that has the answer.
I've been thinking that it might be useful if you could actually _remove_ all the stored facts that the LLM has inside of it. I believe that an LLM that didn't natively know a whole bunch of random trivia facts, didn't know basic math, didn't know much about anything _except_ what was put into the initial query would be valuable. The AI can't hallucinate anything if it doesn't know anything to hallucinate.
How you achieve this practically I have no clue. I'm not sure it's even possible to remove the knowledge that 1+1=2 without removing the knowledge of how to write a python script one could execute to figure it out.
They've got a big database of logical reasoning propositions that they have been trying to do a much more formal-logic process with.
The real solution to these problems is to train transformers on a more human-like information context rather than pure text. Hallucinations should naturally decrease as LLMs become more "agentic"
Is there some "confidence" coefficient that we can extract from AI?
I would claim that hallucinations are required for creativity and problem solving. A "novel" answer is a hallucination to the existing dataset. For a simple example, have ChatGPT-4 come up with a new words that combine two concepts. I imagine this wouldn't be possible if hallucinations weren't allowed.
A simple casual proof:
Emitting a token T for an input I for a given system has 0 entropy and requires knowing the entire state of the system at the time input I is given. This includes knowing the entire system itself, as knowing the state alone is meaningless without having knowledge of the system.
This is of course impossible for a model that is contained within the system emitting tokens T itself, however, an approximation is possible. The bound of approaching 0 entropy necessarily requires learning the inherent dynamics of the system itself. Any model that trivially depends upon statistics could not do causal reasoning, it would become exponentially less likely over time. At long output lengths, practically impossible.
Thus, beyond a certain point, to reduce the entropy any further beyond some softly-defined minima for cross-entropy, the system must inherently generalize to the underlying problem that yields the tokens in question (hence ML's data hungriness).
I don't think I feel surprised by this comment, it is a personal belief after all, but seeing similar ideas posted with such confidence both here and on Twitter is something that I do find personally confusing, it's not really grounded in my view in the information theory of how deep learning works. Even with a purely statistical argument once could make a very strong argument rather easily. Especially comparing small to large models.
This is handwaving. Yes, a system that is fundamentally based on statistics will require more and more data and compute power to be able to continue to function over longer and longer output lengths.
But you don't know a priori what the shape of that curve is, or how far along it current LLMs are (maybe their creators have some idea, but I suspect not even they truly understand where on that curve the current systems are).
Thus, there's no reason to assume that the system is "generaliz[ing] to the underlying problem" at all. And in fact, I'd argue that not only is there no reason to do so, there are strong reasons to assume that it is not doing that.
Edit: I noticed we could be reading this in different directions -- I'm reading the OP's post as treating LLMs as large Markov-chain-memorizing models, which is where the statistics argument comes into play. Heck, the curse of dimensionality alone makes the memorization aspect of it intractable, so there is a sort of compression, only what kind of compression. I agree that statistically it's going to approach what happens in the real world on the text as things get larger, but simple hallucinated chains of text in an autoregressive matter severely breaks the causal regime. I think that was where I was coming from originally, please do let me know if I've misunderstood, however.
Second Edit: w.r.t. to the underlying manifold generalization, yes, this is a natural consequence of the operators used. directly measuring how much it's happening is intractable but by the nature of the system operating under them we are actually somewhere on that compressive spectrum of generalization.
Proving that an AI is reasoning and not hallucinating seems super difficult. Even proving that there's a difference would be difficult. I'm more open to the idea that reasoning in general is just statistical hallucination even for humans, but that's almost off topic.
> Any model that trivially depends upon statistics could not do causal reasoning, it would become exponentially less likely over time. At long output lengths, practically impossible.
It's not clear to me that it _doesn't_ fall apart over long output lengths. Our definition of "long output" might just be really different. Statistics can carry you a long way if the possible output is constrained, and it's not like we don't see weird quirks in small amounts of output all the time.
It's also not clear to me that adding more data leads to a generalization that's closer to the "underlying problem". We can train an AI on every sonnet ever written (no extra tagged data or metadata) and it'll be able to produce a statistically coherent sonnet. But I'm not sure it'll be any better at evoking an emotion through text. Same with arithmetic. Can you embed the rules of arithmetic purely in the structure of language? Probably. But I'm not sure the rules can be reliably reversed out enough to claim an AI could be "reasoning" about it.
It does make me wonder what work has gone in to detecting and quantifying reasoning. There must be tons of it. Do we have an accepted rigorous definition of reasoning? We definitely can't take it for granted.
That is the trouble with mixing inductive reasoning sometimes with a problem that has mathematical roots. There are degrees where it's intractable to easily measure how much something is happening, but we have a clean mathematical framework that answers these questions well, so using it can be helpful.
The easiest example of yours that I can tie back to the math is the arithmetic in the structure of language. You can use information theory to show this pretty easily, you might appreciate looking into Kolmogorov complexity as a fun side topic. I'm still learning it (heck, any of these topics goes a mile deep), but it's been useful.
Reasoning on the other hand I find to be a much harder topic, in terms of measuring it. It can be learned, like any other piece of information.
If I could recommend any piece of literature for this, I feel like you would appreciate this most to start diving into some of the meat of this. It's a crazy cool field of study, and this paper in particular is quite accessible and friendly to most backgrounds: https://arxiv.org/abs/2304.12482
(Full disclosure: I work at Vectara, where this blog was published)
I'm claiming there's a great deal of work in the realm of research that I'm excited by: research I expect to be done by humans. I do expect/am supposing that result may be LLMs of a different nature: to apply guard rails around generative LLM systems, but that's not to anthropomorphize them: just to suppose their purpose.
The term "hallucination" does anthropomorphize LLMs, but I think that's now accepted nomenclature in the industry, at least for the time being, and it's helpful to have some standard nomenclature to describe some of the benefits and problems.
It does not as hallucinations are not only something humans experience. Beyond that it is now an accepted term of art used to describe a specific behavior exhibited by an LLM that is separate from the biological one.
The problem I have with the term is that we already have one that describes much more accurately what these models are doing: it's called guessing. Guessing is simply reporting information one does not know to be true. As a model does not have data points regarding certain information, each token it returns is done so with lower and lower confidence. It's literally guessing. But since we aren't exposed to the confidence score of the completion it's taken to be full confidence, when it is not the case.
To be clear, I'm referring to things like GPT-3.5 reportedly consistently messing up on statements like "what's heavier, two pounds of feathers or a pound of bricks". Being consistently wrong in the same way implies to me (but I don't know for sure) that the class of response is high probability in an absolute sense.
I can't find the article that demonstrated the sort of things that GPT consistently gets wrong, but it was things like common misconceptions and sayings.
So in that case it's not guessing and not wrong; it's indeed producing something that is correct, but still false. Now we're really getting into the weeds here though.
The response to "how much opportunity is still in front of us for adversarial LLM systems that try to detect/control for hallucinations." is by nature infinite or none (as in its futile). As "hallucinations" are whatever the developer deems to be a "hallucination". To hallucinate anthropomorphizes the model to be a human actor and leads "treatment" like a drug to be administered. A physician saying that "oh my patient is hallucinating" they have a mental disorder. This implies that there is a ground truth the developer knows to "not hallucinations". To make a model with such procedures would inherently contain any bias from the development team. Using techniques like Constitutional AI to align models with ethical values, relies on someone making that "ethical value".
Statistical artifacts or general incorrectness in responses are a more accurate to this research. Adopting a "bias mitigation" mindset, viewing bias reduction as an ongoing process of detection and correction, not a one-time fix produces its own errors or inconsistencies is a better solution, as the red tape is out of scope of the model itself. Treat every model as rouge, similar to zero trust of a computer system. If the solution is not also an AI model, then you avoid a sort of Inventors Paradox by dehumanizing people into agents.
Both of these are ideas at the current state of AI is a social dilemma, that people have been warning about for years. The nature of the words we use change our mental model and perception of the tools we create. While history shows it is something in human nature to anthropomorphize items and tools like cars and boats, they do not talk back in a human readable format. If my car started to "hallucinate" I would think I am driving inside a Herby or some other living car. The parallels made between silicon and carbon are similar but profoundly inaccurate to our current understanding, but to go down that path is off topic. As an engineer please do not anthropomorphize your creations it is unhealthy and may lead to superficial relationships. To control "Statistical artifacts" or "hallucinations" is to be the same contextually, and there is always middleware and interface management, but to "hallucinate" changes how one may perceive the ai's functions. Please do not anthropomorphize LLM.
I want to improve the typography - the current feedback I'm getting so far is that the current design works well with two friends who have dyslexia, but I think that's more to do with the formatting than current typography... I have no good sense of typographical style though.
And now the more relevant comment!
Consensus truth can be far from objective truth as ideas don't compete based on value or truth or usefulness: but merely by how sticky, how replicatable in the brains of others.
https://en.wikipedia.org/wiki/Anosognosia
Right now, the AI is operating "inside the bubble" of our thoughts and so is unlikely to figure out our collective blind spot, but once one can interact with the world in more meaningful ways, we should pay really close attention to the mistakes it makes.
Depends on how many connections a statement has. `1+1=2` has an extremely high number of connections, so it would be very hard to vary that to `1+1=3`. When you start to say things like `Daniupolomonotrofin activates the mitoyuicinain leading to weightloss`, very few connections, very easy to vary (lie).
But we have heuristic.
You do not even have the "perfect hammer", for example. But "close enough" is fine.
Use a language model to run queries against another language model and check if it’s hallucinating.
Say we have two language models A and B, A is the verifier and B is being trained.
We give A accesses to a ground truth database, and then we get it to generate questions it knows the answer to based off of its knowledge base.
A asks B those questions and then it verifies Bs output against its knowledge base and we use the veracity of Bs output as the reward function.
So you say okay, I'll leave out some facts from training to use to teach it to say I don't know. Now you have model B that is similar to A but on some things that A answers correctly on, B answers I don't know. ... and on some new facts both A and B hallucinate, so B is strictly worse than A-- they both hallucinate but A knows more.
Using known unknown facts to train for saying "I don't know" is only useful if it produces a general ability to say "I don't know" against unknown unknowns. And I don't know if anyone has managed to demonstrate that result.
It's difficult in general to know what the model does and doesn't know, and LLM isn't a trivia bot-- and it knows tons of stuff that exists nowhere explicitly in the training data (which is why it's useful over and above a verbatim internet search!).
It's fun to play with the boundary of LLM knowledge by conversing with it in ROT13 or asking it write with bizarre constraints and watch its intelligence fall away.
The size of the LLMs is an easy way to currently demonstrate this - even if you took only the factual statements from all its training material and compressed them, they'd be larger than the model. The model isn't magic thus it can't contain all those facts.
The only way it can write a fact is if those words are simply the most likely completions and happen to be right.
This means you'd be essentially training it randomly by selecting factual answers. You wouldn't be reinforcing that it gave you a correct fact, just whatever the sentence structure was that you judged to be factual.
I think what would happen is that it would start to write very careful statements which would be more likely to be technically correct merely by not being wrong.
For example, if you trained by asking questions like what year president George Washington was born it would quickly learn to stop guessing a year because that's got a low probability of being right and those statements would get trained out. It'd probably write something like "Before 1760" because that statement has a much higher likelihood of being right even if it's a less useful answer.
No. Hallucination is any idea which was not assessed for truth. Whatever statement is not put over the "testing table" and analyzed foundationally counts as hallucination.
> ...There was a study where they took a group of actors, had them go out to 200 people and ... they walked up to each person and ... held a cup of coffee, they walk up to you [and hand you the cup], ... look down so you can't say yes or no, and you'd end up taking it - they get their phone, they adjust it, they take [the cup] back and say "Thank you", that's the whole thing - same facial expression for every person, only difference, half got an iced coffee the other got hot coffee. Now 30 minutes go by, they send out ... a research assistant with a clipboard and they come up to these same individuals and say, "If you give us two minutes your time we'll give you twenty dollars: will you just read these three paragraphs and tell us what you think of this character?" ... they read the three paragraphs and they say "What do you think of the main character in this little story?". 81% that were given iced coffee say the person is cold and uncaring; 80% percent (a one percent variance) of those [of the] hot said the person is warm and connected and caring. ... Most people think their thoughts are their thoughts, when really your thoughts have been primed by the environment
Sure, on a cold day a hot coffee may be nice to hold and an iced coffee may be unpleasant. But on a hot day? The hot coffee burns your hands; the iced coffee is refreshing to hold. And this is even accepting the premise of the claim.
(Or perhaps it's reversed! Maybe we'd be more likely to think well of someone who asked us to do something mildly unpleasant, in a Franklin-esque way, and resent someone who offered us a pleasant moment only to take it away.)
(Or maybe the idea is mental association? I do, after all, think well of glassblowers and distrust ice cream vendors.)
Are we unknowingly influenced by little things, by the wrong things? Definitey, but I'd be surprised if this were an accurate example.
edit: my apologies for the bit of snark ^^'
I am tickled a bit by the story, though.
Possibly, but that (that the case mentioned by Tony Robbins is not factual) is not the important thing: it is an example of what is involved in not vetting ideas. Practice ("before accepting an idea, think") that should be a basically held mandate - also (not primarily) given the warnings of a large part of psychology that shows how easy it is to be fooled.
If you want more solid examples, I recommend the speeches from Charlie Munger about "human misjudgement", which take much from Cialdini and consolidated researchers (and have a "been there, seen that" seal from Munger).
Behaviour that is avoidable in humans, we surely do not want in machines. Tools are there to enhance strength, not weakness.
"Before accepting an idea, think": this we "expect" from all.
I think that's a good story, because I think such unjustified 'weights' DO have a profound effect. That it's BS is entirely the point. It's a fable about hallucination. It's part of intelligence.
Yes, but let us be clear about the critical point: humans do have a faculty - whether they use it or not - to check their thought for foundations, for validity. And this faculty is the condition for a legitimacy to make statements.
People may conjure unfounded ideas, immature constructs (this is studied since millennia): but well developed intellects then process those ideas, test them, refine them.
Given that it is relatively easy to be fooled, or dramatically easy (especially since mistakes can have a unbelievably staggering cost), and we are supposed to know that - after psychology, criminology etc., or just simply after the facts that we are more experienced than infants -, the problem of discrimination and reflection remain central and critical.
And (just in case): there is absolutely no need to emulate fools: the goal is the opposite.
One could vary temperature in order to try to avoid wild swings of hallucination but that has downsides as well.