Hacker News Reader

Comment by jampekka | Hacker News Reader

I'm of the opinion that the current architectures are fundamentally ridden with "hallucinations" that will severely limit their practical usage (including very much what the hype thinks they could do). But this article puts an impossible limit to what it is to "not-hallucinate".

It essentially restates well known fundamental limitations of formal systems and mechanistic computation and then presents the trivial result that LLMs also share these limitations.

Unless some dualism or speculative supercomputational quantum stuff is invoked, this holds very much to humans too.

View on HN

> fundamentally ridden with "hallucinations" that will severely limit their practical usage

On the other hand, a LLM that got rid of "hallucinations" is basically just a thing that copy-paste at that point. The interesting properties from LLMs comes from the fact that it can kind of make things up but still make them believable.

As per this article, even copy-paste hallucinate e.g. because there are no infinite datasets.

> can kind of make things up but still make them believable

This is the definition of a bullshitter, by the way.

Or, you know, fiction writer. Some of us like little stories.

Its important that anyone reading fiction knows its fiction.

Without that we can run into a situation where a fictional story on the radio convinces the public that we are under alien attack in what could be called a war of the worlds.

Yes! Or a situation where the public was not under any misconception at all, where there were multiple intermissions throughout the play calling attention to the fact that it was a work of fiction, but someone made up the idea of mass panic because it made for a good story that sold a lot of newspapers!

I would say that it's the responsibility of the reader to read any content in the context of which it's published. For example, to expect a radio play to be fictional as the vast majority of listeners did.

Training with inappropriate data will make copy-paste equally bad with hallucinating. For example, fiction or sarcasm may be taken out of context and used as a serious output.

What's missing is sanity-checking the sources and the output.

C.S. Peirce, who is known for characterizing abductive reasoning and had a considerable on John Sowa’s old school AI work, had an interesting take on this. I can’t fully do it justice, but essentially he held that both matter and mind are real, but aren’t dual. Rather, there is a smooth and continuous transition between the two.

However, whatever the nature of mind and matter really is, we have convincing evidence of human beings creating meaning in symbols by a process Peirce called semiosis. We lack a properly formally described semiotic, although much interesting mathematical applied philosophy has been done in the space (and frankly a ton of bullshit in the academy calls itself semiotic too). Until we can do that, we will probably have great difficulty producing an automaton that can perform semiosis. So, for now, there certainly remains a qualitative difference between the capabilities of humans and LLMs.

I don’t know the technical philosophy terms for this, but my simplistic way of thinking about it is that when I’m “seriously” talking (not just emitting thoughtless cliché phrases), I’m talking about something. And this is observable because sometimes I have an idea that I have trouble expressing in words, where I know that the words I’m saying are not properly expressing the idea that I have. (I mean — that’s happening right now!)

I don’t see how that could ever happen for an LLM, because all it does is express things in words, and all it knows is the words that people expressed things with. We know for sure, that’s just what the code does; there’s no question about the underlying mechanism, like there is with humans.

> I have an idea that I have trouble expressing in words, where I know that the words I’m saying are not properly expressing the idea that I have. (I mean — that’s happening right now!)

That's a quality insight. Which, come to think of it, is an interestingly constructed word given what you just said.

Producing text is only the visible end product. The LLM is doing a whole lot behind the scenes, which is conceivably analogous to the thought space from which our own words flow.

You can simulate a NAND gate using balls rolling down a specially designed wood board. In theory you could construct a giant wood board with billions and billions of balls that would implement the inference step of an LLM. Do you see these balls rolling down a wood board as a form of interiority/subjective experience? If not, then why do you give it to electric currents in silicon? Just because it's faster?

Tangentially, Peirce has a diagrammatic logical system that's built entirely on conjunction and negation which is isomorphic to propositional logic. He also defined extensions for what we now call predicate logic and modal logic.

John Sowa annotated Peirce's tutorial and it's quite interesting[1].

[1] https://www.jfsowa.com/pubs/egtut.pdf

Your point of disagreement is the _medium_ of computation? The same point can be made about neurons.

Do you think you could have the same kind of cognitive processes you have now if you were thinking 1000x slower than you do? Speed of processing matters, especially when you have time bounds on reaction, such in real life.

Another problem with balls would be the necessity of perception, that you can't really do with balls alone, you need different kind of medium for perception and interaction, that humans, (and comuputers) do have.

Are you familiar with Searle's work[1] on the subject? It's fun how topical it is here. Anyhow maybe the medium doesn't matter, but the burden of proof for that claim is on you, because it's contrary to experience, intuition, and thought experiment.

[1] https://plato.stanford.edu/entries/chinese-room/

Really out-of-ignorance: Is 'proof' the right word here? A more substantial philosophical counter-argument may be needed, but proof sounds weird in these "metaphysical" (for now) discussions.

You raise a valid point. Proof is an overloaded term with differing meanings in metaphysics, math, and law. However in all three cases it's obviously different attempts to grasp at the same ultimate thing: truth.

My take on Searle is that he was a hack. It's possible I judge too harshly, that _I_ am a hack (the likeliest, tbh) or that I and his writing have some fundamental life outlooks different.

Regardless, I think the Chinese room experiment is bunk and proves nothing. And I fail to gather where the medium of computation steps in the Chinese room experiment. The "computer" might as well be a bunch of neurons in a petry dish.

I guess the proof will be in the pudding when we develop superhumanly intelligent AI.

> I guess the proof will be in the pudding when we develop superhumanly intelligent AI.

I'm not sure that's the case. The universe itself is already capable of superhuman intelligence. There's nobody alive that can predict how wind will flow over an airfoil better than a wind tunnel.

The actual proof will be in the pudding if we develop superhumanly creative AI.

> Your point of disagreement is the _medium_ of computation?

No. My point is that we should not impute interiority onto computation. The medium is a thought experiment meant to stimulate your intuition that computational sophistication does not imply interiority. Unless you do think that the current state of balls rolling down a board does entail a conscious experience?

To adjust the speed, just imagine the balls rolling down in 100000000x time. I don't know where you stand, but I still don't think there is a conscious experience on the board.

Yes, but that space is entirely derived from human expressions, in words, of their own thought space. The LLM has no direct training access to the humans’ thoughts like it does to their words. So if it does have comparable thought space, that would imply such a space can be reconstructed accurately after passing through expression in words, which seems like a unsupported claim based on millennia of humans having trouble understanding each others’ thoughts based on verbal communication, and students writing essays that are superficially similar to the texts they’ve read, but clearly indicate they haven’t internalized the concepts they were supposedly learning.

It’s not to say there couldn’t be a highly multimodal and self-training model that developed a similar thought space, which would be very interesting to study. It just seems like LLMs aren’t enough.

What impresses me is frankly how bad it is.

I can't claim to have tried every model out there, but most models very quickly fail when asked to do something along the lines of "describe the interaction of 3 entities." They can usually handle 2 (up to the point where they inevitably start talking in circles - often repeating entire chunks verbatim in many models), but 3 seems utterly beyond them.

LLMs might have a role in the field of "burn money to generate usually-wrong ideas that are cheap enough to check in case there's actually a good one" though.

Reply on news.ycombinator.com