Hacker News Reader

Comment by HarHarVeryFunny | Hacker News Reader

Are we really still having this conversation in 2024 ?! :-(

Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.

The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.

View on HN

Yes, they are auto-completers, but they are auto-completers that are layered AND operate in higher dimensional spaces. This throws all intuitions off, and I think makes it misleading to think of them as "just" auto-completers. That's part of the story, but not the whole of it.

I suspect we are much closer to auto-completers than most of us like to think, but we're also trained+incentivized by culture, education, parenting, socializing, to produce "useful" results.

Maybe part of the problem is in the data set: how much modeling of "how to admit ignorance or uncertainty" are in LLMs training data sets? If you read the internet, all you see is confident replies to other confident replies. Ignorance or non-confidence tends to elicit either a bluff or non-response. If you read technical literature, you see much of the same.

Maybe LLMs are trained on a dataset, and thereby inherit a culture that's accidentally biased toward ignorant confidence. In human conversation, if somebody asks a question and I don't know the answer, I say I don't know. On the internet, I just skip it and leave it for somebody else who thinks they know.

All this is to say: maybe a statistical autocompleter can admit ignorance instead of firing "neural noise" based on barely-there loose associations. Maybe it just needs a stronger pathway toward talking about not knowing when there's not a strong association.

Wasn't it that it by design prefers things expressed with certainty?

One problem is that the LLM's own knowledge, or lack of it, doesn't follow from any individual training sample(s). Even if there were a bunch of "I don't know X" samples in the training set, that ought to be trumped by one authoritative "X is ..." one.

The next problem is that the LLM doesn't know how reliable it's various training sources are (unlike a human who might trust personal experience > textbook > twitter comment), or even which samples come from which source so that it could learn that.

I have thought a lot about this and I suspect "I don't know" would be devastating to the model.

The magic is in the fact that the model can't say "I don't know". The model would have to have arbitrary thresholds and a type of domain/context classification in order to set this arbitrary threshold for the conversation in order to say "I don't know". It would create all these unsolvable boundary conditions. AGI in this context then would be minimizing the need for "I don't know" until it no longer applies. Is that possible? I don't know :)

I would defer to Chomsky also on the subject that we are basically nothing like these models when it comes to language and we are not auto-completers.

I do not find this view to be useful.

The are cases from my personal experience, where I asked somewhat esoteric practical questions, that likely do not (seem) to have a clear answer in the web and I have got considerable help from ChatGPT. At some point this dichotomy of 'statistical word generator' vs 'true intelligence' should go away as it's just not useful. (I think these discussions always lead to Chinese Room problem; and IMO at some point it does not matter what 'dumb' process is behind, provided it solves a problem or the system behaves like an intelligent agent)

Except there is a qualitative difference in the class of knowledge that a statistical word generator and an expert system would generate.

Just because a LLM _can_ offer valuable and insightful information, doesn't mean that it doesn't also hallucinate. The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.

Yup exatly. They are dream machines. LLMs without other systems can only work in the flow. The fact that in this word flow the larger LLMs can generate navigation instructions for actual mazes and solve random algorithmicly generated problems doesn't mean they are not hallucinating it just means we're getting wonderfully useful hallucinations.

Hallucination is a term that means "imagined facts", so it's very hard for me to parse this comment into something meaningful beyond "if we say it always generates hallucination, we can say it always generates hallucinations"

You Google a restaurant that appears to be open. You go there, and you find that the restaurant is no longer there.

Did Google "hallucinate" a restaurant? Because this is no different.

We can empirically test if hallucination is a good word for communicating this concept, by checking if people describe(d) that as a hallucination (they don't).

This is all IMHO, I'm not trying to be difficult or nitpick, I just don't understand the idea as communicated. As applied to LLMs, it sounds like hallucination == could be wrong, and this Google example seems further away even when steel-manning, ex. we don't say all Google results are hallucinated.

It doesn’t mean automatically wrong. It’s just bullshitting. It makes up something that fits a pattern. Depending on the question, the pattern may be right more often than not.

If you ask ChatGPT, “hey is the McDonald’s near my house open at 6pm?” It doesn’t know anything about where you are or if there’s a McDonald’s or what its hours are. It will likely hallucinate that sure, it’s open at 6pm. But when it does so is it “right” in a meaningful way?

Yup. IMHO, I think "bullshitting" is a much better word than hallucinating and/or getting it right!

Much like real life bullshitters, it is inclined to say something truthful-sounding, but doesn't actually have a strong reliability towards truth per se.

Bullshitting implies intent to deceive. As far as we know, an LLM honestly "believes" (as if you need another rabbit hole) what it says. Delusion, perhaps?

Delusion implies a degree of consistency, though. LLMs can be on point one minute and completely off the rails the next even when prompted with the same prompt. Hallucination fits better here as it speaks to the real-time "perception" (there's another one for you).

An LLM is not a brain, though, so no matter which analogy you choose, it will come with some flaws. Regardless, "hallucination" has moved past analogy territory and now has its own LLM-specific usage with reasonably wide acceptance so the analogy angle is now moot anyway.

>Bullshitting implies intent to deceive

Not in the formal sense. The philosopher Harry Frankfurt famously distinguished bullshitting from lying because a liar knows the truth and is trying to hide it where a bullshitter is simply trying to sound convincing and may or may not be telling the truth (and may not even know themselves if they are)

https://en.wikipedia.org/wiki/On_Bullshit

In the current formal sense. It may be true the formal sense in 1986 was different. Words do evolve in meaning over time, but since we're talking about right now...

You are right that lying and bullshitting are different. A lie is a false statement with intent. Bullshit is nonsense with intent. A false statement and nonsense may share some similarities, but are ultimately different.

Perhaps nonsense is the word we should be applying to LLMs, but often what they say isn't nonsense, even if only by accident, so that doesn't exactly work either. Regardless, it doesn't matter now. As before, "hallucination" has moved beyond analogy and now has its own LLM-specific usage.

You can ask it „Is the McDonalds nearby open? I live in Brumbledon, Ohio.“ It will do a search, and then confidently state that the McDonalds in Brumbledlon, Ohio is in fact open 24/7. I guess it doesn’t matter that no such town exists.

That's fair. I'm honestly stunned how little work there's been to incorporate search, in a real way, into products. perplexity, ChatGPT and bing to some extent, that's it.

(Screenshot is my yet to be released flutter app)

https://imgur.com/a/HCVEfkQ

That's not what the GP is arguing, though

> Except there is a qualitative difference in the class of knowledge that a statistical word generator and an expert system would generate.

There's a lot of difference between the two, and you don't have to treat it as one or another. It's OK to treat it as something in the middle.

> The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.

Completely agree, sans the word "troubling". It's not troubling. It is what it is. As long as you keep it in mind when you use it, and treat it as an entity that can be completely wrong, and use it where it's OK to be completely wrong (e.g. when the output is easily verifiable), there's nothing "troubling" with that.

You're stuck on a problem. You grab a random comic book from the shelf and something that is written in the comic book sparks your solution in your head.

How intelligent is the comic book? Is it hallucinating or being correct or what?

The answer is none of those things, right? YOU did the thinking. Not the inanimate object; what it did was a happy coincidence.

Pointless discussion on semantics.

If a random page in a random comic book gives me the answer I seek 30% of the time, it's incredibly useful. Little effort was spent in seeking the answer, and the 70% of the time it is wrong led to little waste in time.

Now if you put a mechanical arm interface in the middle where I give my query to a machine, and it randomly picks the comic book and page, which answers my question 30% of the time - I have no trouble calling it "intelligent".

Contrast it with Google searches that don't give me the answer I seek, but use up an order of magnitude more of my time.

It's like you want to protect our right to hallucinate, because ChatGPT can't.

Can you provide some examples of things that aren’t on the web but that ChatGPT helped you with? I’ve yet to see an example that’s not in the likely training set (which includes more than just the public Internet).

But to your point, I disagree: the mechanism matters. Just because you haven’t detected the limitations of the mechanism behind ChatGPT doesn’t mean it’s not there.

My guess is he's referring to things that are out there on the web, but not in the top results and not easy to find.

Perhaps I'm confused, but your question seems contradictory and ambiguous. You first imply (I think?) that ChatGPT is limited to web-sourced information, but then acknowledge the training set includes more than just the public web. Can you please clarify what you are asking?

This is just admitting you don't care about knowledge or truth.

Sure, but the question posed isn't whether LLMs exhibit intelligence (obviously so, minimally in Chinese Room sense), or can they combine sources (sure, no way to stop them), but why do they hallucinate.

Notwithstanding the amazing things they can do, I don't think it helps understanding by viewing LLMs in too abstract of a way as intelligent agents. After all, in reality they are "just" language models, and hopefully in 2024 the nuance of what they needed to learn to be GOOD language models doesn't need to be explicitly stated every time we discuss them.

Looking at them as language models, it's easy to explain why they hallucinate, are poor reasoners, etc, and IMO does nothing to distract from understanding why they also exhibit intelligence when operating "in distribution".

I've never been worried about LLMs. I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.

I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.

The problem isn't the people. It's the tech companies.

The tech companies are telling people that it's intelligent, and the tech companies are using it to answer people's questions as if they're presenting facts.

People are using it the way they're told.

If you advertise something as a solution, don't be surprised when people use it to solve things.

Always remember companies are people too... until we make companies powered by AI. Till that day the underlying problem is always with the people.

Why is this concern more important the what people interpret from the media, social media and the dissemination of information in general where lies and fabrications are also commonplace? Like surely people will always fall for nonsense, lies or fabrications and there is nothing that can be done about that.

Because LLMs strip away all the context surrounding the information it spits out that let you evaluate its trustworthiness. They're incredibly useful tools, I use them constantly when coding but I can do that because I know enough to validate the information and it happens that the cost of validating the output with the docs is shorter than reading them to find the relevant functions.

I wouldn't dare try to use an LLM for a chemistry question because I wouldn't be able to tell if it makes any sense or not. But if you're not a "tech person" and all you see is some company advertising their AIs as magical knowledge engines with disclaimer text that wouldn't pass accessibility tests, why wouldn't you assume they know their stuff? The Perplexity ads are bordering on negligent.

The difference is that web/social media is branded as an intelligent being you can ask any question of. We all agree the web is _also_ not reliable, but many people will think GPT / Gemini are verifiably accurate when they aren’t.

> We all agree the web is _also_ not reliable

You need to expand your circle a bit more :-)

As with all these discussions: accountability and consequence.

We can point at a media company, call out its vested interests, scream about its bias, protest in front of its office, sue it for slander and misrepresentation. We can call out individual personalities the same way. We can strive to drive the companies out of business and the personalities out of work, if we deem it necessary, and we can accumulate a paper trail that holds each one to account.

As neither individuals nor corporate entities, algorithms do not yet carry this kind of legal or public accountability even as we some start to hold them up as oracles. In most cases, failures of an algorithm are treated simply as bugs or user mistakes. Nobody is responsible for anything bad and the so the algorithm can persist and its vendor can shrug off their own responsibility by gesturing towards an perpetual development process instead of accepting consequence: "we work to make the algorithm better every day, try again tomorrow!"

>We can point at a media company, call out its vested interests, scream about its bias, protest in front of its office, sue it for slander and misrepresentation.

Right but the previous election had Russian servers spinning up fake news websites that displayed straight up generated news. Again how do you hold them accountable? You can't the only defence against bullshit is independent thinking.

Not much different. Except that there's a dumb thing that humans are doing which is giving weight to the magic brainy computers.

All that training data would make even grep look smart. I'm just glad those magical forest creatures made the data in the first place.

I've found that people in general seem to trust computers more than humans, which made sort of sense for a while.

What they don't fully realize is that this is a completely different game; now the computer is just guessing, as opposed to following a deterministic algorithm to the answer.

And this misunderstanding carries the potential for pretty serious consequences, good luck getting that loan once a computer finds some arbitrary pattern and says no.

If only it would tell you "You've criticized the war effort that day in 2004", in stead it will do parallel construction. The end game will be a kind of SEO for human profiles and we will live happily ever after by the best practice guide lines.

This is the most prescient point I have read hitherto regarding LLMs. We are fashioning for ourselves gods of wood and stone.

s/LLM/Wikipedia/g

Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion:

LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words).

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734

LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-...

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

Yes, because just like chess ELO that we discussed the other day, they need to learn this in order to do well on their training objective - impersonating (continuing) their training sources. If they are continuing a lie then they need to have recognized the input as having this "context", and take that into account during prediction.

Right but then the problem of hallucination has little to do with statistical generation and much more the utter lack of any incentive in pre-training or otherwise to push features the model has already learnt into the words it generates.

Right, more due to the inherent nature of an LLM than due to that nature being a statistical generator, although as such they amount to the same thing.

One way of looking at it is model talking itself into a corner, with no good way to escape/continue, due to not planning ahead...

e.g. Say we ask an LLM "What is the capital of Scotland?", and so it starts off with an answer of the sort it has learnt should follow such a question "The capital of Scotland is ...". Now, at this point in the generation it's a bit late if the answer wasn't actually in the training data, but the model needs to keep on generating, so does the best it can and draws upon other statistics such as capital cities being large and famous, so maybe continues with "Glasgow" (a large famous Scottish city), which unfortunately is incorrect.

Another way of looking at it rather than talking itself into a corner (and having to LLM it's way out of it), is that hallucinations (non-sequiturs) happen when the model is operating out of distribution and has to combine multiple sources such as the expected form of a "What is .." question reply, and a word matching the (city, Scottish, large, famous) "template".

I think this may be the best explanation I've seen on the topic!

But, shouldn't that situation be handled somewhat by backtracking sampling techniques like beam search? But maybe that is not used much in practice due to being more expensive.. don't know.

Thanks!

I'm not sure if beam search would help much since it's just based on combining word probabilities as opposed to overall meaning, but something like "tree of thoughts" presumably would since I doubt these models would rate their own hallucinated outputs too highly!

Asking "why do large language models hallucinate?" is a good question to ask and answer, just like "why do birds sing?" or "why is the sky blue?" are. The problematic part is when you're somehow surprised that the sky is blue, birds sing and LLMs do nothing but hallucinate.

Tiger gotta hunt, bird gotta fly, LLM gotta answer "why, why, why?"

Tiger gotta sleep, bird gotta land, LLM gotta produce a token from the context at hand

I think we're all having different experiences and conversations with LLMs, and sometimes framing them purely as "statistical word generators" or expecting them to function like AI in the traditional sense of automated problem-solving might not capture the whole picture.

Like many in this community, I use LLMs daily. My main use case now isn’t software development but rather getting assistance in connecting concepts that I don’t know precisely, guided by my intent. In some ways, it's more akin to "out-of-the-box thinking" but with a tool that helps me explore ideas I might not reach on my own or suffer within the economy of search [1][2]. I might know something about X, Y, and Z, but there's a concept W that ties them all together. Without using W, people in that field might not grasp the connections between X, Y, and Z. Apologies for the abstraction here. I suppose it's my epistemological bias showing!

Even if LLMs don't precisely connect the dots, they help my brain connect them faster.

[1] https://en.wikipedia.org/wiki/Search_theory

[2] https://www.di.ens.fr/~lelarge/soc/varian2.pdf

"Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ?"

That's exactly the question the paper attempts to answer: why do LLMs ever get it right? The answer is that on topics where there's a lot of data and a general consensus on what the right answer is, the statistical model will find that answer, and otherwise you get junk. That's why they work so well for people trying to write Python or Javascript, for example.

But I already knew this, you might say. Sure, but the authors produced evidence to back it up.

There's really no "right or wrong" per se -- the question that's really being asked is "to what extent does it resonate with a person?"

As context sizes grow, it's easier to add lots of information outside of training data via in-context learning, which should offset that issue quite a bit.

> Why would a language model do anything other than "hallucinate"

Indeed, why?!?!?! Why do they so often get the correct answer to very direct questions? Saying "it's in the training data" -- I dare you to find anything in the training data that talks about how many "q's" there are in the word "mortuary", and yet it "hallucinates" up answers to this.

> without any care

What does it mean to care? What question could you ask of an LLM that would allow you to assess how much it "cares" about something?

> but it's not like they have a choice of not answering or saying "I don't know" in other cases

Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?

There seems to be a dichotomy of reactions to LLMs. There are technical people saying "it's just an autocomplete engine reciting things from its training set" and there are non-technical people saying "it does more than just next token completion, it is trained to use language".

The first group is technically correct but ignores the fact that it can, emergently, do things far outside of an explainable capability, the second group is technically incorrect but correctly perceives that it can use language in novel ways.

Neither group captures the fact that we built extremely complex linear algebra machines that for reasons we do not understand, despite being trained on an incredibly simple task (next-token-prediction) are capable of actually using language in a way that ten years ago we assumed only humans could do.

> Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?

No, but when it does occur in the training data it's a reflection of that particular source/speaker not knowing, which isn't the same as the LLM not knowing because it was also trained on millions/billions of additional sources.

For an LLM to learn to say "I don't know" appropriately, it would need to know when it itself doesn't know (and have that change if you told it), and of course it doesn't have that capability.

> of course it doesn't have that capability

Yes, of course not. Of course. There's no way, for example, that I could ask it if it knew what number I was thinking of, and, after I tell it the number and ask the same question, that it could express that it didn't know before and did know after. Absolutely impossible. Out of the question that it would have this capability. Clearly impossible given its architecture. No way that it could possibly do this task. Why, it would require advances in machine learning and quantum computers and understanding a theory of consciousness and perception at a level that we won't have for centuries. Maybe even a completely new design and training procedure to even begin to approach this insurmountable task.

And if it did demonstrate this ability, then clearly your assessment of its capabilities would be completely wrong and you would have to step back and reconsider how much its training data reflects its abilities.

Well, so far there's no model that knows what it doesn't know, and hallucination continues to be a problem.

So maybe you can enlighten us all, AI labs included, with your genius as to how to solve hallucination, and how to do so only via changes to the training set since that is what you suggest.

To make things easy for you, lets assume that every training text has been augmented with source information such that the model could potentially learn which sources are trustworthy on given subjects or not, and therefore assess whether it knows something or not.

So what else are you going to add to the training set that you claim would induce it to learn this self-referential "I know X" knowledge to better achieve it's next word loss ? Why do you think the AI labs have not done what you are suggesting ?

You misunderstand me -- my position is that we are completely ignorant as to how LLMs do what they do, and it is delusional to think that we understand what their capabilities are or how they are limited.

Until we understand why they sometimes don't hallucinate we can't begin to approach the problem of why the do. And above I directly refute the notion that no model "knows what it doesn't know" -- they are manifestly capable of demonstrating ignorance.

As to next steps in this I have no concrete thoughts, and I don't even know whether LLMs are a dead end. What I suspect is that as we apply more computational power to the problem, as we've seen in the past, computational power plus network depth plus training data means that the structure of the models (like the attention mechanism or the tokenization) will eventually be unnecessary and the structures will be self-discovered and more capable for it.

As models get deeper and more expressive I am confident that things like a broader capability to understand the limitations of its own knowledge or to test its knowledge against objective reality will emerge organically. We may be able to make leaps by creating shortcuts for some of these mechanisms, but in the end depth and computation power will trump all.

> my position is that we are completely ignorant as to how LLMs do what they do

That's simply wrong.

Whole teams of very smart people at places such as Anthropic are working on this exact problem (the field of "mechanistic interpretability"), and have made considerable headway, and have published their results.

Just a note: Truthiness means a feeling of truthfulness, even if not actually true. I'd argue that LLMs do care about that, but I suspect you meant 'truthfulness'.

> Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.

That's a false dichotomy.

Their ability to generate facts is a consequence of the word statistics they are truly using. I think it's fair to say the statistics explanation is a more accurate interpretation.

More accurate than what? Statistics "explanation" is something which is technically correct. But it also doesn't present the full picture — for example, the fact that LLMs clearly build an internal mental model of the question they're talking about.

I can offer cite that hallucinations are an innate property of LLMs, can you provide one that shows they have 'an internal mental model'?

https://arxiv.org/abs/2401.11817

Sure! https://thegradient.pub/othello/

Uhm.

I don't really know if using an "experiment" is the right way to test this?

Like, if you want to test such hypothesis you should first prove that these LLMs were created with that specific function in mind, we are not observing a natural phenomena, it is a manmade(manmade...loosly) algorithm .

You don't know the internal architecture of the system and you are testing in a reality narrow situation, this is basically the scientific method applied incorrectly.

We are using different meanings for terms.

Building a world model for a perfect information game is different than building an mental model of the external world.

In context learning is a well known property of LLMs, while real world generalization, often described through the common sense problem is not.

To me, 'mental models' are personal, internal representations of external reality, which LLMs currently lack, being limited to the corpus.

I believe they used the game to show how the same underlying technology would build a mental model not because the game was perfect-information, but because it was easy to probe for without a lot of other unrelated concepts getting in the way.

Perhaps this is what you're looking for (similar technique, larger model?) https://www.anthropic.com/news/golden-gate-claude

> In the “mind” of Claude, we found millions of concepts that activate when the model reads relevant text or sees relevant images, which we call “features”.

> One of those was the concept of the Golden Gate Bridge. We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.

Sounds like a mental model to me! An internal representation of an external concept which exists in the real world.

No, you're not. Are you genuinely trying to suggest that LLMs, which can:

- Construct arbitrary text that isn't just grammatically but semantically coherent

- Derive intent, subtle intent, from user queries and responses

- Emulate endless different personalities and their reactions to endless stimuli

- Describe in detail the statics and dynamics of the world, including sight, smell, touch and sound

do not have a model of the external world? What do you think a "corpus" means in this context? How is the "corpus" of sensory and evolutionary data that makes you up in any way different?

LLMs are excellent common sense reasoners, and they generalize just fine. Why exactly do you think they get things _subtly_ wrong? Make up API syntax that looks sensible but isn't actually implemented? In order to make these guesses they need to have generalized, they need an understanding of the structure underlying naming, such that they can produce _sensible_ output even if they lack the hard facts.

You are correct. We are flooded with studies on AI now, so can't find reference.

But just few months ago, saw example of AI, from video, building an internal representation of the world. An internal model of the world. Everyone saying this can't be done, it already is. Maybe can argue it wasn't an LLM, and then I'd say were nitpicking over which technology can do it or not. We already have example of tying them together, symbols and LLM's.

I'm not convinced at all. The only thing they are doing is perturbing some of the model weights in intermediate layers, and seeing if the output of the final layer is consistent with the perturbations. It would be a shitty model if that was not the case.

The fancy part in the paper is figuring out how to perturb the intermediate layers in the way you want. But the findings are not impressive.

Note also that the "probe geometry" stuff is so speculative they left it out of the academic paper completely.

In the same way, it has been known since the 90s that if you take the matrices from Finite Element Models and visualize them as graphs, structures appear that kind of resemble the physical appearance of the object being modelled. Here for instance is for a helicopter:

http://yifanhu.net/GALLERY/GRAPHS/GIF_SMALL/Pothen@commanche...

Yet nobody thinks Finite Element Models have an internal mental representation of the world.

"Yet nobody thinks Finite Element Models have an internal mental representation of the world."

At this point, I'm not sure some wouldn't argue that.

The difference is, put the AI on a loop, with constant feedback, learning. Instead of just a 'pre-trained' model. Make the actual model, live, always learning, so the context window is infinite. This of course would not be for everyone, because it would take all the resources of the training infrastructure to be focused on one person/view. But that gets closer to the human mind, and at that point, we probably couldn't say for sure that the 'perturbations' aren't experiencing something subjective.

Where is the proof that humans have an internal mental representation of the world.

They build internal representations of the input only, and to the extent needed, to get the statistics right. This really isn't a "world model" of factual data, but rather a "source model" of what would various sources (training texts) say.

The responses of the model don't represent what it understands per some internal model, because there is no "it", only models of the sources it was trained on, and it'll just as happily generate lies as truths, or smart vs dumb answers (it's all just words) if that is what its source modelling calls for.

What most people mean when they say the model is hallucinating/bullshitting isn't where it has learnt a lie, but rather where is is operating "out of distribution", and is therefore (unknowingly) generating a mashup from multiple only loosely related/matching source contexts.

I don't even like the word "hallucination", because it's an anthropomorphism, further suggesting/misleading into an impression that LLMs have anything to do with how a human would reason.

Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.

The use of "hallucinate" is also my gripe.

Philosophy of language, and Wittgenstein in particular, has more specific words for the output insofar as they mean anything to humans: "senseless" or "nonsense".

But I don't think those terms are accurate. If I say "John Doe was born in 1987", that's a perfectly sensible sentence, even if I don't actually know when he was born and even if it turns out he was actually born in 1971. A nonsense sentence would be something like "John Doe cat five green explicate" or "h4stga3jkui7rutjdyrst".

I don't understand your point.

You've just re-presented one difference between sense and nonsense in philosophy of language. We might call that first phrase wrong or an error or inaccurate if that was the case. We don't need to use a word like "hallucinate" for your second phrase --- we can just call it nonsense.

"Hallucinate" is far more inaccurate and confusing. It paints a false picture that melds 2 distinct things, as you've described, which need not happen at the same time: errors and nonsense.

I think the term forst originated with the dream-like images of google deepmind, these were really similar to human visual hallucinations, i guess the term stuck

It's unfortunate, but it is what it is.

When a human hallucinates, that is due to the brain operating out of spec. When an LLM "hallucinates", it's performing exactly as intended, it just has delivered a nonsensical result which is the consequence of the tech behind the LLM.

You aren't going to like this, but don't insult the hammers.

Instead of "hallucinating" I would have preferred the term "bullshitting" -- in the Harry G. Frankfurt sense of not caring about the truth of one's utterances. But it's too late for that.

https://link.springer.com/article/10.1007/s10676-024-09775-5

Using "bullshit" would be interesting, but to me would introduce a backdoor anthropomorphism to describe the output. The picture is still too human.

Isn't Frankfurt's concept of bullshit made up of 2 parts: 1) a distinction between lying and telling the truth, AND 2) the absence of caring about either when speaking when normally it's assumed present?

Part 1 seems to apply, but part 2 wouldn't. It doesn't make sense to talk about GPT "caring" about its output beyond anthropomorphism. No one talks about their computer caring about having correct or accurate output and neither is it assumed. People would think your imagining a demon in the box. Really, it's even odd to say "GPT lied" outside of very specific circumstances.

I think "bullshitting" fits better than "hallucinating" - just keep spitting out words rather than admit ignorance, but maybe the best human analogy is freestyle rapping where one has to keep the flow of words coming regardless!

Maybe we just need to coin a new word for it - "LLM-ing" perhaps ?!

""Are we really still having this conversation in 2024 ?! :-( ""

</Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".

As opposed to what?

added sarcasm note. it was referring to the parent.

I dont get it, what's the unique insight in the article, isn't it just stats and bad data?

It's just being a bit reductive to just toss AI out as un-interesting because it is just based on statistics.

And, think there is question of 'bad data' so study is bad/invalid, versus studying impact of 'bad data' on responses is the result.

I meant, the answer to the question of why does AI hallucinate is the same answer to the question "why does any statistical system ever produce an incorrect result" which I thought was well established.

How can it be well established when the LLMs are still actively changing and improving.

Don't think we can say, statistical systems can have incorrect results, hence we no longer need to study how statistical system produce incorrect results, because we already know they can have incorrect results.

I am impressed that with multimodal modals in the hands of everyone people still say this. I mean it’s true on the surface but the ability to take a picture of a data sheet and wiring diagram and get it to give me step by step instructions to wire two components together and an example of the UART protocol in cpp that while not directly functional captures all the essential information I need. That’s amazing and it’s not just word prediction, the source is images of documents with diagrams!

Anyone who does this and comes away jaded has lost the ability to dream.

It's not just true on the surface—it's true at its core. All those features you mention are based on statistical relationships. This doesn't mean that these systems can't be very useful, but it also doesn't mean that they are _intelligent_, as much as we like to call them that. They have no understanding of their input or output, beyond being able to pattern match and mechanically decide what the next token should be based on their training data.

Then you need to explain why statistics isn't enough to be intelligent. We are at the point where that isn't an obvious argument anymore and I'll point to the best models as to why you might be wrong.

Because being able to find patterns in large amounts of data will never make the system intuitively understand math[1], or history[2], or any other field. It will always depend on the data and biases we feed it, at least with the current approaches.

You might say that this is how humans learn and demonstrate intelligence, but it's not the same thing. We have the ability to advance our understanding of the universe without being explicitly trained on every topic. Until we can build systems that can do this, I wouldn't label them as intelligent.

But, again, this doesn't mean that they can't be useful.

[1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-high...

[2]: https://www.theguardian.com/technology/2024/mar/08/we-defini...

I never said it wasn’t statistical at any level so I’m not sure what your point is. My point was almost everything we do is an approximation (I mean, all of science and technology and engineering) and very little of it isn’t. Fake it until you make it is pervasive in human endeavors. Over time the errors and issues will be smoothed away until it’s still there but it’s not noticeable enough to be a major problem.

It being statistical doesn’t make it useless. It not being some metaphysical concept of awareness doesn’t make it useless.

But the multimodal model support puts the lie to it being just a stochastic parrot of language. The domains it works in -is- more abstract than syntax and grammar even if it’s grounded in it. That’s part of the entire power of the model. In the end it leads to a predicted token but the path it takes is more subtle and complex than a simple Markov chain of tokens.

> I never said it wasn’t statistical at any level so I’m not sure what your point is.

You said it was true on the surface, and I'm pointing out that it's true at its core. Your amazement at how well it works is based on our inability as humans to find the same patterns in the data as these models do. Ascribing some higher level sense of intelligence or understanding to these systems because of this party trick is anthropomorphizing what they actually are. We would all be better served by keeping in mind how they work, instead of being surprised when they do output the wrong pattern.

> Over time the errors and issues will be smoothed away until it’s still there but it’s not noticeable enough to be a major problem.

Why do you think this is guaranteed? We can keep throwing data and compute at these systems, and while we might continue to get better results, the current systems will never intuitively understand that e.g. 9.11 is lower than 9.2[1], unless those specific examples are in their training data. As we reach the limits of the data we can feed them, and we have to generate synthetic data, there's no reason to believe that the current approaches will ever fix these problems at their core.

> It being statistical doesn’t make it useless. It not being some metaphysical concept of awareness doesn’t make it useless.

I never said it was. I agree that this can be useful, but again, as long as we're aware of their limits. It's good keeping this in mind as we're reaching the peak of inflated expectations of the hype cycle.

[1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-high...

Reply on news.ycombinator.com