Hacker News Reader

Comment by lolinder | Hacker News Reader

> By establishing the mathematical certainty of hallucinations, we challenge the prevailing notion that they can be fully mitigated

Having a mathematical proof is nice, but honestly this whole misunderstanding could have been avoided if we'd just picked a different name for the concept of "producing false information in the course of generating probabilistic text".

"Hallucination" makes it sound like something is going awry in the normal functioning of the model, which subtly suggests that if we could just identify what went awry we could get rid of the problem and restore normal cognitive function to the LLM. The trouble is that the normal functioning of the model is simply to produce plausible-sounding text.

A "hallucination" is not a malfunction of the model, it's a value judgement we assign to the resulting text. All it says is that the text produced is not fit for purpose. Seen through that lens it's obvious that mitigating hallucinations and creating "alignment" are actually identical problems, and we won't solve one without the other.

View on HN

Yes, exactly, it’s a post-facto value judgment, not a precise term. If I understand the meaning of the word, “hallucination” is all the model does. If it happens to hallucinate something we think is objectively true, we just decide not to call that a “hallucination”. But there’s literally no functional difference between that case and the case of the model saying something that’s objectively false, or something whose objective truth is unknown or undefinable.

I haven’t read the paper yet, but if they resolve this definition usefully, that would be a good contribution.

Exactly this, I've been saying this since the beginning. Every response is a hallucination - a probabilistic string of words divorced from any concept of truth or reality.

By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences. Therefore, creating something that imitates a truthful sentence will often happen to also be truthful, but there is absolutely no guarantee or any function that even attempts to enforce that.

All responses are hallucinations. Some hallucinations happen to overlap the truth.

Ok, but I think it would be more productive to educate people that LLMs have no concept of truth rather than insist they use the term "hallucinate" in an unintuitive way.

I don't know about OP, but I'm suggesting that the term 'hallucinate' be abolished entirely as applies to LLMs, not redefined. It draws an arbitrary line in the middle of the set of problems that all amount to "how do we make sure that the output of an LLM is consistently acceptable" and will all be solved using the same techniques if at all.

LLMs do now have a concept of truth now since much of the RLHF is focused on making them more accurate and true.

I think the problem is that humanity has a poor concept of truth. We think of most things as true or not true when much of our reality is uncertain due to fundamental limitations or because we often just don't know yet. During covid for example humanity collectively hallucinated the importance of disinfecting groceries for awhile.

> humanity collectively hallucinated the importance of disinfecting groceries for awhile

I reject this history.

I homeschooled my kids during covid due to uncertainty and even I didn't reach that level, and nor did anyone I knew in person.

A very tiny number who were egged on by some YouTubers did this, including one person I knew remotely. Unsurprisingly that person was based in SV.

It's not some extremist on YouTube, disinfecting your groceries was the official recommendation of many countries worldwide, including most of Europe. I couldn't say how many people actually followed the recommendation , but I would bet it's way more than a tiny number.

This is the first I’ve even heard of people disinfecting their groceries because of Covid. Honestly that sounds rather crazy to me.

There was a period near the start of the pandemic, especially while the medical establishment was trying to avoid ordinary people wearing masks in order to help stockpile them for high priority workers, when a lot of emphasis was put on surface contact.

If it's extremely important to wear gloves and keep sanitizing your hands after touching every part of the supermarket, it stands to reason that you'd want to sanitize all of the outside packaging that others touched with their diseased hands as soon as you brought it into your house. Otherwise, you'd be expected to sanitize your hands every time you touched those items again, even at home, right?

Of course, surface contact is actually a very minor avenue of infection, and pretty much limited to cases where someone has just sneezed or coughed on a surface that you are touching, and then putting your hand to your nose or maybe eyes or mouth soon after. So sanitizing groceries is essentially pointless, since it only slightly reduces an already very small risk.

I did not do this personally but I know a number of people (blue state liberal city folk) I don’t think it was that unusual.

I think taking decisions based on different risk models is not a hallucination.

To the extreme: if during covid someone would live completely off grid (no contact with anyone) would have greatly reduced infection risk, but I would have found the risk model unreasonable.

The problem with LLM-s is that they don't "model" what they are not capable off (the training set is what they know). So it is harder for them to say "I don't know". In a way they are like humans - I seen a lot of times humans preferring to say something rather than admitting they just don't know. It ss an interesting (philosophical) discussion how you can get (as a human or LLM) to the level of introspection required to determine if you know or don't know something.

Or to put it more concisely, LLMs behave similar to a superintelligent midwit.

Exactly, we think of reasoning as knowing the answer but the real key to the enlightenment and age of reason was admitting that we don't know instead of making things up. All those myths are just human hallucinations.

Humans taught themselves not to hallucinate by changing their reward function. Experimentation and observation was valued over the experts of the time and pure philosophy, even over human-generated ideas.

I don't see any reason that wouldn't also work with LLMs. We rewarded them for next-token prediction without regard for truth, but now many variants are being trained or fine-tuned with rewards focused on truth and correct answers. Perplexity and xAI for example.

> LLMs do now have a concept of truth now since much of the RLHF is focused on making them more accurate and true.

Is it? I thought RLHF was mostly focused on making them (1) generate text that looks like a conversation/chat/assistant (2) ensure alignment i.e. censor it (3) make them profusely apologize to set up a facade that makes them look like they care at all.

I don't think one can RLHF the truth because there's no concept of truth/falsehood anywhere in the process.

If people already understand what "hallucination" means, then I think it's perfectly intuitive and educational to say that, actually, the LLM is always doing that, just that some of those hallucinations happen to coincidentally describe something real.

We need to dispell the notion that the LLM "knows" the truth, or is "smart". It's just a fancy stochastic parrot. Whether it's responses reflect a truthful reality or a fantasy it made up is just luck, weighted by (but not constrained to) its training data. Emphasizing that everything is a hallucination does that. I purposefully want to reframe how the word is used and how we think about LLMs.

In other words, all models are wrong, but some are useful.

Precisely, I'm glad someone picked up on the reference!

I think you're going too far here.

> By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences.

It's not a "total coincidence". It's the default. Thus, the model's responses aren't "divorced from any concept of truth or reality" - the whole distribution from which those responses are pulled is strongly aligned with reality.

(Which is why people started using the term "hallucinations" to describe the failure mode, instead of "fishing a coherent and true sentence out of line noise" to describe the success mode - because success mode dominates.)

Humans didn't invent language for no reason. They don't communicate to entertain themselves with meaningless noises. Most of communication - whether spoken or written - is deeply connected to reality. Language itself is deeply connected to reality. Even the most blatant lies, even all of fiction writing, they're all incorrect or fabricated only at the surface level - the whole thing, accounting for the utterance, what it is about, the meanings, the words, the grammar - is strongly correlated with truth and reality.

So there's absolutely no coincidence that LLMs get things right more often than not. Truth is thoroughly baked into the training data, simply because it's a data set of real human communication, instead of randomly generated sentences.

Strongly correlated != 100% overlap

A 99% overlap can still be coincidence.

And even if it was ‘absolutely no coincidence’, that is still only reflective of the reality as perceived by the average of all the people from the training set.

The problem - as defined by how end users understand it - is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence that it does accurate information.

That's how you end up with grocery store chatbots recommending mixing ammonia and bleach for a cocktail, or lawyers using chatbots to cite entirely fictional case law before a judge in court.

Nothing that comes out of an LLM can be implicitly trusted, so your default assumption must be that everything it gives you needs verification from another source.

Telling people "the truth is baked in" is just begging for a disaster.

> your default assumption must be that everything it gives you needs verification from another source

That depends entirely on what you're doing with the output. If you're using it as a starting point for something that must be true (whether for legal reasons, your own reputation as the ostensible author of this content, your own education, etc.) then yes, verification is required. But if you're using it for something low-stakes that just needs some semblance of coherent verbiage (like the summary of customer reviews on Amazon, or the SEO junk that comes before the recipe on cooking websites which have plenty of fiction whether or not an LLM was involved) then you can totally meet your goals without any verification.

People have been capable of bullshitting at scale for a very long time. There are occasional consequences (hoaxes, scams, etc.) but the guidelines around fide sed vide are ancient; this is just the latest addendum.

This is just moving the goalposts. The post I replied to was claiming that models "have the truth baked in". Real people in the real world are misusing them, in no small part because they don't know that the models are unreliable, and OP's claims only make that worse.

> is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence

which is a good model for what humans do as well

> It's not a "total coincidence". It's the default. Thus, the model's responses aren't "divorced from any concept of truth or reality" - the whole distribution from which those responses are pulled is strongly aligned with reality.

One big caveat here - the responses are strongly aligned with the training data. We can't necessarily say that the training data itself is strongly aligned with reality.

> Even the most blatant lies, even all of fiction writing, they're all incorrect or fabricated only at the surface level - the whole thing, accounting for the utterance, what it is about, the meanings, the words, the grammar - is strongly correlated with truth and reality.

I would reject this pretty firmly. As you said, people write whole novels about imagined worlds and people about magic or technology or whatever that doesn't or can't exist. The LLM may understand what words mean and "know" how to string them together into a meaningful and grammatical sentence, but that's entirely different than a truthful sentence.

Truth requires some mechanism of fact finding, or chains of evidence, or admitting when those chains don't exist. LLMs have nothing like that.

So much truth here, very refreshing to see!

About time too, the sooner we can stop the madness the better, building a society on top of this technology is a movie I'd rather not see.

Maybe they shouldn’t have mixed truthful data with obviously untruthful data in the same training data set?

Why not make a model only from truthful data? Like exclude all fiction for example.

That wouldn't prevent hallucination. An LLM doesn't know what it doesn't know. It will always try to come up with a response that sounds plausible, based on its knowledge or lack thereof.

1) It's impossible to get enough data to train one of these well while also curating it by hand.

2) Even if you could, randomly sampling from a probability distribution will cause it to make stuff up unless you overfitted on the training data. An example that's come up in thread is ISBNs—there isn't going to be enough signal in the training set to reliably encode sufficiently high probability strings for all known ISBNs, so sometimes it will just string together likely numbers.

maybe hallucination is all cognition is, and humans are just really good at it?

In my experience, humans are at least as bad at it as GPT-4, if not far worse. In terms, specifically, of being "factually accurate" and grounded in absolute reality. Humans operate entirely in the probabilistic realm of what seems right to us based on how we were educated, the values we were raised with, our religious beliefs, etc. -- Human beings are all over the map with this.

> In my experience, humans are at least as bad at it as GPT-4, if not far worse.

I had an argument with a former friend recently, because he read some comments on YouTube and was convinced a racoon raped a cat and produced some kind of hybrid offspring that was terrorizing a neighborhood. Trying to explain that different species can't procreate like that resulted in him pointing to the fact that other people believed it in the comments as proof.

Say what you will about LLMs, but they seem to have a better basic education than an awful lot of adults, and certainly significantly better basic reasoning capabilities.

> Trying to explain that different species can't procreate like that resulted in him pointing to the fact that other people believed it in the comments as proof.

Those two species can't interbreed apparently, but considering the number of species that can [1] produce hybrid offspring, some even from different families, it is reasonable to forgive people for entertaining the possibility.

[1] https://en.m.wikipedia.org/wiki/List_of_genetic_hybrids

I don't think it's remotely reasonable. The list you refer to, which I don't need to click on as I'm already familiar with it, is animals within the same family, e.g. bi cats.

Raccoons are not any type of feline, and this should be basic knowledge for any adult in any western country who grew up there and went to school.

There are at least a couple of examples in the article that you refuse to read that describe hybrids from different families. Sorry, but your purported basic knowledge is wrong.

I'm not 'refusing to read' it, I said I'm familiar with it because I've read it numerous times in the past.

Which examples are you referring to? The only real example seems to be fish.

In any case I was using 'family' in a loose sense, not in the stricter scientific biological hierarchy sense.

My basic knowledge is not wrong at all, because my point was that animals that far apart could not reproduce. That's it. The wiki page you linked doesn't really justify your idea that because some hybrids exist people might think any hybrid could exist.

The point is, it's frankly idiotic or at least extremely ignorant for anyone 40 years of age who grew up in the US or any developed country to think that.

I also very much doubt the people who believe a racoon could rape a cat and produce offspring are even aware of that wiki page or any of the examples on it. Hell, I doubt they even know a mule is a hybrid. Your hypothesis doesn't hold water.

Additionally, most of the examples on that page are the result of human intervention and artificial insemination, not wild encounters. Context matters.

Ok but humans aren't being hyped as this incredible new tech that's a going to lead to the singularity.

This is demonstrably not true. People also bullshit, a lot, but nowhere near the level of an LLM. You won't get fake citations, complete with publication year and ISBN, in a conversation with a human. StackOverflow is not full of down voted answers of people suggesting to use non-existent libraries with complete code examples.

Both of those terms have precise meanings. They're not the same thing. Summarized --

Cognition: acquiring knowledge and understanding through thought and the senses.

Hallucination: An experience involving the perception of something not present.

With those definitions in mind, hallucination can be defined as false-cognition that is not based in reality. It's not cognition because cognition grants knowledge based on truth and hallucination leads the subject to believe lies.

In other words, "humans are just really good at hallucination" rejects the notion that we're able to perceive actual reality with our senses.

I mean hallucination in the context of this conversation: probabilistic token generation without any real knowledge or understanding.

Maybe if we add a lot of neurons and make it all faster, we would end up with “knowledge” as an emergent feature? Or maybe we wouldn’t.

It's definitely part of what cognition is, hallucinogens/meditation/etc allows anyone to verify that much.

Intuitively cognition is several systems running in tandem, supervising and cross checking answers, likely iteratively until some threshold is reached.

Wouldn't surprise me if expert/rule systems are up for some kind of comeback; I feel like we need both, tightly integrated.

There's also dreams, and the role they play in awareness, some kind of self reflective work is probably crucial.

That being said, I'm 100% sure there is something in self awareness that is not part of the system and can't be replicated.

I can observe myself from the outside, actions and reactions, thoughts and feelings; which begs the question: who is acting and reacting, thinking and feeling, and what am I if not that?

Humans can hallucinate but later determine that what they thought was occurring was not actually real. LLMs can't do that. What you're saying sounds to me rather like what some people are tempted to do on encountering metaphysics: posing questions like "maybe everything is a dream and nothing we experience is real". Which is a logically valid sentence, I guess, but it really is meaningless. The reason we have words like "dreaming" and "awake" is that we have experienced both and know the difference. Ditto "hallucinations". It doesn't seem that there is any difference to LLMs between hallucinations and any other kind of experience. So, I feel like your line of reasoning is off-base somewhat.

I agree. I shouldn't have used the word "hallucinations" since the point of the conversation above my comment was that they are not really hallucinations by any meaningful definition of the word.

My question was more about whether "babbling" with statistically likely tokens can eventually emerge into real cognition. If we add enough neurons to a neural network, will it achieve AGI? or is there some special sauce that is still missing.

Exactly. Even a broken clock is right twice a day.

Excellent point. I did not think about hallucination in this manner before.

I don’t know who/how the term was initially coined in this context, but I’m concerned that the things that make it inaccurate are also, perhaps counterintuitively, things that serve the interests of those who would overstate the capabilities of LLMs, and seek to cloud their true nature (along with inherent limitations) to investors and potential buyers. As you already pointed out, the term implies that the problems that are represented are temporary “bugs”, rather than symptoms of the underlying nature of the technology itself.

How different things would be if the phenomenon had been called "makin' stuff up" instead. Humans make stuff up all the time, and make up far more outrageous things than AIs make up. One has to ask whether humans are really intelligent /not entirely sarcasm.

I'd prefer the phenomenon be called "saying the first thing that comes to your mind" instead, because humans do that a lot as well, and that happens to produce pretty much the same failures as LLMs do.

IOW, humans "hallucinate" exactly the same way LLMs do - they just usually don't say those things out loud, but rather it's a part of the thinking process.

See also: people who are somewhat drunk, or very excited, tend to lose inhibitions around speaking, and end up frequently just blurting whatever comes to their mind verbatim (including apologizing and backtracking and "it's not what I meant" when someone points out the nonsense).

Confabulation is the term I’ve seen used a few times. I think it reflects what’s going on in LLMs better.

Yes, I thought this was already the agreed upon term for those in the know. I don't know what all this HN hullabaloo is about.

This is due to an entire field of AI /machine learning leaning into anthropomorphism shaping terminology reinforced by a narrative.

Which is totally on them. We had to suffer a decade of being told the human brain is a neural net (as if my brain was a numeric matrix, freaking ridiculous) they get to suffer when that dumb analogy comes back to bite them.

It’s still a bug, even if it’s the only way the system can behave as currently designed. I agree that “hallucination” is a poor term for it, though. For medication, we call a bug a “side effect” even though it’s really just a chemical interaction which, given enough information, could be predicted.

Ultimately the computer only does what we tell it to do. That has always been the case and probably always will be, just as we are the result of our inputs.

As with most bugs, I think to solve hallucinations we will need to better understand the input and its interactions within the system.

> It’s still a bug

Only if it's actually possible to remove this behavior. I don't think there's any evidence of that at this point.

I find it helpful to distinguish between bugs and design flaws.

A bug is caused by a poorly implemented version of the design (or a literal bug in the system). Fixing a bug requires identifying where the system varies from the design and bringing it into alignment with the design.

A design flaw is a case where the idealized system as conceived by the engineers is incapable of fully solving the problem statement. Fixing a design flaw may require small tweaks, but it can also mean that the entire solution needs to be thrown out in favor of a new one.

Importantly, what's a design flaw for one problem statement may be just fine or even beneficial for another problem statement. So, more objectively, we might refer to these as design characteristics.

Hallucinations are a special case of two low-level design characteristics of LLMs: first, that they are trained on more data than can reasonably be filtered by a human (and therefore will be exposed to data that the humans wish it weren't) and second, that they produce their text by sampling a distribution of probabilities. These two characteristics mean that controlling the output of an LLM is very very difficult (or, as the article suggests, impossible), leading both to hallucinations and alignment concerns (which are actually the same concept framed slightly differently).

If the problem statement for an LLM application requires more 99% factual accuracy or more than 99% "doesn't produce content that will make investors nervous" accuracy, these design characteristics count as a design flaw.

99% is way too low for production use.

If that's true for your use case (it's not true for all) then yes, including LLMs in your design would be a design flaw until they get much much better.

It's OK for cases, when humans are worse and/or much more expensive.

I think there's a useful distinction between plausible-seeming text that is wrong in some subtle way, vs text that is completely fabricated to match a superficial output format, and the latter is what I wish people used "hallucination" to mean. A clear example of this is when you ask an LLM for some sources, with ISBNs, and it just makes up random titles and ISBNs that it knows full well do not correspond with reality. If you ask "Did you just make that up?" the LLM will respond with something like "Sorry, yes, I made that up, I actually just remembered I can't cite direct sources." I wonder if this is because RLHF teaches the LLM that humans in practice prefer properly formatted fake output over truthful refusals?

How does a model "know full well" that it output a fake ISBN?

It's been trained that sources look like plausible-titles + random numbers.

It's been trained that when challenged it should say "oh sorry I can't do this."

Are those things actually distinct?

In fairness, they will also admit they were wrong even if they were right.

This comment should be pinned at the top of any LLM-related comment section.

It should be part of every AI related story on the news. Just like they keep saying "X, formerly Twitter".

Nah it's quite pedantic to say that 'this neologism does not encapsulate the meaning it's meant to'

This is the nature of language evolution. Everyone knows what hallucination means with respect to AI, without trying to confer to its definition the baggage of a term used for centuries as a human psychology term.

Neologism undersells what this term is being used for. It's a technical term of art that's created its own semantic category in LLM research that separates "text generated that is factually inaccurate according to ${sources}" from "text generated that is morally repugnant to ${individuals}" or "text generated that ${governments} want to censor".

These three categories are entirely identical at a technological level, so I think it's entirely reasonable to flag that serious LLM researchers are treating them as distinct categories of problems when they're fundamentally not at all distinct. This isn't just a case of linguistic pedantry, this is a case of the language actively impeding a proper understanding of the problem by the researchers who are working on that problem.

> This is the nature of language evolution.

Only if it sticks. Hallucination is such an unnatural term for the phenomenon I would be surprised to see it stick.

> Everyone knows what hallucination means with respect to AI

This is false. It took me months to realize this just meant "output incoherent with reality" rather than an issue with training—the natural place for perceptual errors to occur.

"Hallucination" is derogatory and insulting when aimed at normal people who hear normal voices which don't necessarily belong to corporeal beings, or originate in the natural world. Labeling "hallucinations" and pathologizing, then medicating them, constitutes assault and bigotry.

"Hallucination" applied to inanimate and non-sentient software is insulting and presumptive on a different level. I'm fine with "confabulation".

How sure are you that humans don’t also “simply produce plausible-sounding text”?

I’m watching my toddler learning to speak and it’s remarkably like an early LLM that outputs gibberish with nuggets of sense.

Conversely I’ve seen middle-aged professionals write formal design documents that are markedly inferior to what GPT-4 can produce. This is at every level: overall structure, semantics, and syntax. Diagrams with arrows the wrong way, labels with typos, labels on the wrong object, mixed up icons, tables with made-up content, and on and on.

Aren’t we all doing this to some degree? Half way through a sentence do you known ahead of time exactly what you’re about to say three pages later? Or are you just “completing” the next word based on the context?

> How sure are you that humans don’t also “simply produce plausible-sounding text”?

I'm not, and I didn't say I was.

Nothing in my comment is downplaying the success and utility of LLMs. Nothing in my comment is suggesting that they won't get better or even that they won't get better than us humans. It is strictly an observation that these two things that we call different names—hallucinations and alignment—are actually equivalent and can each be expressed in terms of the other. A hallucination is a specific kind of misalignment, nothing more.

I agreed with you until your last sentence. Solving alignment is not a necessity for solving hallucinations even though solving hallucinations is a necessity for solving alignment.

Put another way, you can have a hypothetical model that doesn't have hallucinations and still has no alignment but you can't have alignment if you have hallucinations. Alignment is about skillful lying/refusing to answer questions and is a more complex task than simply telling no lies. (My personal opinion is that trying to solve alignment is a dystopian action and should not be attempted.)

My point is that eliminating hallucinations is just a special case of alignment: the case where we want to bound the possible text outputs to be constrained by the truth (for a value of truth defined by $SOMEONE).

Other alignment issues have a problem statement that is effectively identical, but s/truth/morals/ or s/truth/politics/ or s/truth/safety/. It's all the same problem: how do we get probabilistic text to match our expectations of what should be outputted while still allowing it to be useful sometimes?

As for whether we should be solving alignment, I'm inclined to agree that we shouldn't, but by extension I'd apply that to hallucinations. Truth, like morality, is much harder to define than we instinctively think it is, and any effort to eliminate hallucinations will run up against the problem of how we define truth.

Your argument makes several mistakes.

First, you have just punted the validation problem of what a Normal LLM Model ought to be doing. You rhetorically declared hallucinations to be part of the normal functioning (i.e., the word "Normal" is already a value judgement). But we don't even know that - we would need theoretical proof that ALL theoretical LLMs (or neural networks as a more general argument) cannot EVER attain a certain probabilistic distribution. This is a theoretical computer science problem and remains an open problem.

So the second mistake is your probabilistic reductionism. It is true that LLMs, neural nets, and human brains alike are based on probabilistic computations. But the reasonable definition of a Hallucination is stronger than that - it needs to capture the notion that the probabilistic errors are way too extreme compared to the space of possible correct answers. An example of this is that Humans and LLMs get Right Answers and Wrong Answers in qualitatively very different ways. A concrete example of that is that Humans can demonstrate correctly the sequence of a power set (an EXP-TIME problem), but LLMs theoretically cannot ever do so. Yet both Humans and LLMs are probabilistic, we are made of chemicals and atoms.

Thirdly, the authors' thesis is that mitigation is impossible. It is not some "lens" where mitigation is equal to alignment, in fact one should use their thesis to debunk the notion that Alignmnent is an attainable problem at all. It is formally unsolvable and should be rendered as a absurd as someone claiming prima facie that the Halting Problem is solvable.

Finally, the meta issue is that the AI field is full of people who know zip about theoretical computer science. The vast majority of CS graduates have had maybe 1-2 weeks on Turing machines; an actual year-long course at the sophomore-senior level on theoretical computer science is Optional and for mathematically mature students who wish to concentrate in it. So the problem arises is a matter of a language and conceptual gap between two subdisciplines, the AI community and the TCS community. So you see lots of people believing in very simplistic arguments for or against some AI issue without a strong theoretical grounding that while CS itself has, but is not by default taught to undergraduates.

> You rhetorically declared hallucinations to be part of the normal functioning (i.e., the word "Normal" is already a value judgement).

No they aren't: When you flip a coin, it landing to display heads or tails is "normal". That's no value judgement, it's just a way to characterize what is common in the mechanics.

If it landed perfectly on its edge or was snatched out of the air by a hawk, that would not be "normal", but--to introduce a value judgement--it'd be pretty dang cool.

You just replaced 'normal' with 'common' to do the heavy lifting, the value judgment remains in the threshold you pick.

Whereas OP said that "hallucinations are part of the normal functioning" of the LLM. I contend their definition of hallucination is too weak and reductive, that scientifically we have not actually settled that hallucinations are a given for LLMs, that humans are an example that LLMs are currently inferior - or else how would you make sense of Terence Tao's assessment of gpt01. It is not a simplistic argument of LLMs are garbage in garbage out, therefore they will always hallucinate. OP doesn't even show they read or understood the paper which is about Turing machine arguments, rather OP is using simplistic semantic and statistical arguments to support their position.

I didn't say that. I said "hallucination" is a value judgment we assign to a piece of text produced by an LLM, not a type of malfunction in the model.

If we're going to nitpick on word choice let's pick on the words that I actually used.

The term "hallucinate" may not be the best. However, if the context is a device generating a plausible sounding but non-existent scientific reference for a given claim it makes, some words more specific than "producing false information..." seem needed.

These thing make assertions, some true, some false. Since they emulate human behavior, they say things that would tend to convince people of their assertions. So "anthropomorphism" is a complicated question.

True. Actually, researchers know about it. In a sense, i feel "hallucination" is a way to keep the hype up, that we could fix it in the future, given enough data and compute and money. On the other hand, "generating false information by design" is too strong and negative. At least that's what I hear around in my research community.

"Hallucinations" just means that occasionally the LLM is wrong.

The same is true of people, and I still find people extremely helpful.

Except the LLM didn't deliver a "wrong" result, it delivered text that is human readable and makes grammatical sense. Whether or not the information contained in the text is "wrong" is subjective, and the reader gets to decide if it's factual or not. If the LLM delivered unreadable gibberish, then that could be considered "wrong", but there is no "hallicinating" going on with LLMs. That's an anthropomorphism that is divorced from the reality of what an LLM does. Whoever called it "hallucinating" in the context of an LLM should have their nerd credentials revoked.

Human beings have a tendency to prefer comforting lies over uncomfortable truths.

"The truth may set you free, but first it's really gonna piss you off." -G.S.

Human readable, makes grammatical sense, and wrong. And no, that's often not subjective.

It's very subjective. An LLM could return statements like "Global warming is real and man-made", and it also could produce a result like "Global warming is a hoax", and it's definitely up to the reader as to whether the LLM is "hallucinating". It doesn't matter how readable or grammatically correct the LLM is, it's still up to the reader to call bullshit, or not.

If you ask about opinions, sure. Because there are no "true" opinions.

If you ask about the capital of France, any answer but "Paris" is objectively wrong, whether given by a human or LLM.

Paris has not always been the capital of France. Many other cities around France have been capital.

https://en.wikipedia.org/wiki/List_of_capitals_of_France

There's practically no subject you could bring up that an LLM wouldn't "hallucinate" or give "wrong" information about given that garbage in -> garbage out, and LLMs are trained on all the garbage (as well as too many facts) they've been able to scrape. The LLM lacks the ability to reason about what century the prompt is asking about, and a guess is all it is programmed to do.

Also, if you ask 100 French citizens today what the true capital of France is, you're not always going to get "Paris" as a reply 100 times.

But if you ask "what is the capital of France" (not what it has been, but what it is), there is actually only one correct answer. "Capital" has a definition, and the headquarters of the government of France has a definite location. Sure, some French citizens will give a different answer. Some people will say the earth is flat, too. They are wrong.

But we're talking about LLMs in this thread, and the example I used of French citizens not always saying Paris is the capital of France is just an example of how topics can be subjective. If you have something pertinent to the LLM discussion, then please reply.

The capital of France is not subjective. People say stuff. Some of it is subjective, and some of it is just wrong.

So, was your comment about Paris about the LLM discussion, or wasn't it? Because you're the one who brought it up, so if we got off topic, blame yourself.

I have asserted that the LLMs are sometimes flat-out wrong. You have answered that point with an example of... what were you trying to say? That humans can also be wrong? If so, that's true, but so what? We were talking about LLMs. Or were you trying to say that even something like the capital of France is actually subjective? If so, you're wrong. "The capital of France" has only one correct answer, even if there are other answers in the training data.

Or are you trying to say that it's not the LLM's fault, because the wrong answers are in the training data? That's true, but it's irrelevant. The LLM is still giving wrong answers to questions that have objectively correct answers. The LLM had that in its training data; that's what LLMs are; but that doesn't actually make the answer any less wrong.

So what's your actual argument here? You seemed to be headed toward a "no answer can actually be objectively correct", which is both lousy epistemology, and completely unworkable in real life. But then you seemed to veer into... something. What are you actually trying to claim?

This is sounding kind of harsh, but I really am not following what your actual point is.

>>The capital of France is not subjective. People say stuff. Some of it is subjective, and some of it is just wrong.

>"The capital of France" has only one correct answer, even if there are other answers in the training data.

"Champagne is the Champagne capital of France". "Bordeaux is the red wine capital of France". See how easy it is? You're pedantry is only proving that you're a pedant and can't accept anything but you being the only one who is correct. Ease up, bro. We can both be right.

But none of that is about LLMs, I'm just proving a separate point.

>So what's your actual argument here?

A system that's programmed to generate plausible sounding text is "right" when it generates plausible sounding text. It's not "hallucinating", it's not "lying", it's not "wrong", it is operating exactly as designed, it was never programmed to deliver "the truth". It's up the the reader to decide if the output is acceptable, which is entirely subjective to what the reader thinks is right. If the LLM says "Bordeaux is the red wine capital of France" are you going to shit on it and say it's somehow "wrong"? NO ITS WROOOONG THERE CAN ONLY BE ONE TRUE CAPITAL OF FRANNNNNNCEEEE!!! Go ahead and die on that hill if you must.

If this were another website, I'd have blocked you already, because this is entirely a waste of my time.

For the first half of what you said: I will note that "wine capital of France" is a completely different claim than "capital of France", even if many of the words are the same. For the rest: I'll just leave this here for everyone else to judge which of us is being the pedant, and which is arguing just to keep arguing.

As for the second half: I am almost in agreement with your overall point here. LLMs are plausible text generators. Yes, I'm with you there. But LLMs are marketed as more than that, and that's the problem. They're marketed by their makers as more than that.

This is not a technical problem, it's a marketing problem. You can't yell at people for accusing a plausible text generator of "hallucinating", when they were sold it as being more than just a plausible text generator. (The were sold it as being "AI", which is something that you might realistically be able to accuse of hallucinating.) The LLM creators have written a check that their tech, by its very nature, cannot cash. And so their tech is being held to a standard that it cannot reach. This isn't the fault of the tech; it's the fault of the marketing departments.

>But LLMs are marketed as more than that, and that's the problem. They're marketed by their makers as more than that.

The new snake oil, same as the old snake oil. This is no different than any other tech bubble. Nobody paying attention should think otherwise. I don't care how it's marketed, I mean half the US is going to vote for a serial rapist conman thanks to some twisted marketing. People are idiots and are easily fooled, and this has gone on as long as there have been humans. I'm not sure what to say about "marketing".

So finally we can sort of agree on something. But I still think you're giving the LLMs too much credit in suggesting that they will always infallibly say "Paris" when asked where is the capital of France. There's simply no mechanism for the LLM to understand "Paris" or "France" or "Capital". If I asked the LLM that question 1,000,000 times, do you really think it would result in "Paris" 1,000,000 times? I kind of doubt it.

The problem is with the person who is expecting truth from an LLM. So far I don't really see too many people putting absolute faith in anything an LLM is telling them, but maybe those people are out there.

No, I never said that an LLM would always say "Paris". I said that Paris is the actual correct answer. I don't give LLMs that kind of credit; I'm not sure what I said that made you think that I do.

This got real tedious...

lame response.

I said often not subjective. When it says that there are two "r"s in "strawberry", it is wrong, and it is not subjective at all. There's no wiggle room. It's wrong.

The term "hallucination" is in response to what the LLMs are being marketed and sold as what they are supposed to do, not how they work technologically.

It makes sense to have better terms for technical discussions, but this term is going to stick around for mainstream use.

People constantly make this mistake, so just to clarify: absolutely nothing about what I just said implies that llms are not helpful.

Having an accurate mental model for what a tool is doing does not preclude seeing its value, but it does preclude getting caught up in unrealistic hype.

Yeah, I was trying to agree with you :)

But unlike LLMs, a person is capable of reflecting and realizing that they made a mistake. They are also capable of correcting so that they don't make the same mistake again. This is why fallible humans are still useful, and fallible LLMs are not.

Humans can also spend an entire lifetime (or more, across multiple generations) being absolutely, inexorably and violently certain they are correct about something and still be 100% wrong.

I am not disagreeing that either people or LMMs are not extremely helpful in many or most instances. But if the best we can do with this technology is to make human-comparable mistakes WAY faster and more efficiently, I think as a species we’re in for a lot more bad times before we get to graduate to the good times.

"So we built a blind-guessing machine, but how can we tweak it so that its blind guesses always happen to be good?"

If the model starts concocting nonexistent sources, like "articles from serious newspapers that just never existed", it is definitely a malfunction for me. AFAIK this is what happened in the Jonathan Turley case.

It’s also problematic to suggest that the problem is unsolvable. It’s unsolvable inside the LLM. It’s 100% solvable within the product.

Agreed. Put simply, everything produced by an LLM today is an hallucination. Errors are simply poor hallucinations.

Consequently we also shouldn't use intelligence or thinking when dealing with AI

Maybe with vanilla LLMs, but new LLM training paradigms include post-training with the explicit goal of avoiding over-confident answers to questions the LLM should not be confident about answering. So hallucination is a malfunction, just like any overconfident incorrect prediction by a model.

I still think OP has a point. The LLMs evolved after public use to be positioned as oracles which know so much knowledge. They were always probabilistic content generators, but people use them the way they use search engines, to retrieve info they know exists but don't exactly know.

Since LLMs aren't designed for this there's a whole post process to try to make them amenable to this use case, but it will never plug that gap

> but it will never plug that gap

They don't have to be perfect, they just have to be better than humans. And that seems very likely to be achievable eventually.

To be better than humans they have to able confidently say "I don't know" when the correct answer is not available[1]. To me this sounds like a totally different type of "knowledge" than stringing words together based on a training set.

[1] LLMs are already better than humans in terms of breadth, and sometimes depth, of knowledge. So it's not a problem of the AI knowing more facts.

> To me this sounds like a totally different type of "knowledge" than stringing words together based on a training set.

We're desperate to keep seeing ourselves as unique with key distinguishing features that are unreproducible in silicon, but from my long experience with computer chess, every step along the way, people were explaining patiently how computers could never reproduce the next quality that set humans apart. And it was always just wishful thinking, because computers eventually stopped looking silly, stopped looking like they were playing by rote, and started to produce "beautiful" chess games.

And it will happen again, after the next leap in AI, people will again latch on to whatever it is that AI systems still lack, and use it to explain how they'll "always" be lacking... only to eventually be disappointed again that silicon can in fact reach that height too.

Humans aren't magic. Whatever we can do, silicon can do too. It's just a matter of time.

Umm, is this true? Tons of worthless technology is better than humans at something. It has to be better than humans AND better than existing technology.

The only time the LLM can be somewhat confident of its answer is when it is reproducing verbatim text from its training set. In any other circumstance, it has no way of knowing if the text it produced is true or not, because fundamentally it only knows if it's a likely completion of its input.

Post training includes mechanisms to allow LLMs to understand areas that they should exercise caution in answering. It’s not as simple as you say anymore.

Cannot agree enough. Hallucination is a terrible name for this behavior.

Would second what you stated

It's an inaccuracy.

I prefer the clearer term "bullshit"

Reply on news.ycombinator.com