Hacker News Reader

Comment by drbig | Hacker News Reader

Honest question: Why so many people attribute "thinking", "knowing, "understanding", "reasoning", "extrapolating" and even "symbolic reasoning" to the outputs of the advanced token-based probabilistic sequence generators, also known as LLMs?

LLMs are inherently incapable of any that - as in mechanically incapable, in the same way a washing machine is incapable of being an airplane.

Now my understanding is that the actual systems we have access to now have other components, with the LLM being the _core_, but not _sole_ component.

Can anybody point me to any papers on those "auxiliary" systems?

I would find it very interesting to see if there are any LLMs with logic components (e.g. Prolog-like facts database and basic rules that would enforce factual/numerical correctness; "The number of humans on Mars is zero." etc.).

View on HN

Because they don't distinguish between properties of the output and properties of the system which generated it. Indeed, much of the last decade of computer-science-engineering has basically been just insisting that these are the same.

An LLM can generate output which is indistinguishable from a system which reasoned/knew/imagined/etc. -- therefore the "hopeium / sky is falling" manic preachers call its output "reasoned" etc.

Any actual scientist in this field isn't interested in whether measures of a system (its output) are indistinguishable, they're interested in the actual properties of the system.

You don't get to claim the sun goes around the earth just because the sky looks that way.

Do submarines swim? No, but they are faster underwater than all swimmers. Therefore they are the best swimmers despite being unable to swim....

LLMs are producing human level reasoning in many domains, therefore they are the best at reasoning despite being unable to reason...

This whole debate hangs on the definition of "reasoning"

Scientists are extremely interested in measurable results of experiments. I think you are thinking of philosophers.

Can an airplane fly? Can a submarine swim?

Yes, AI may be constructed quite differently from human intelligence. Can it accomplish the same purposes? For some purposes, the answer is a resounding yes as can be seen from its applications around the world by millions of people.

Can an animal ‘think’, ‘understand’, or ‘reason’? Maybe not as well as a homo sapiens. But it’s clear that a raven, a dolphin, or a chimp can do many things we assume require intelligence. (A chimp may even have a slightly larger working memory than a human, according to some research.)

Wouldn’t it be a little preposterous to assume that a young species like ours stands at the pinnacle of the intelligence hierarchy among all possible beings?

> Wouldn’t it be a little preposterous to assume that a young species like ours stands at the pinnacle of the intelligence hierarchy among all possible beings?

You’re right, AI doesn’t need to be AGI to be useful. Most SEO content on the internet is probably even worse than ChatGPT can do. And LLM could hallucinate another Marvel movie since they’re so similar.

My problem is that people make ungrounded claims about these systems either already having sentience or being just few steps away from it. It’s a religion at this point.

some prompts results are only explainable if chatgpt has the ability to produce some kind of reasoning.

As for your analogy, I'm not sure we know enough about human intelligence core mechanisms to be able to dismiss NN as being fundamentally incapable of it.

The reasoning occurred when people wrote the text it was trained on in the first place; it's training data is full of the symptoms of imagination, reason, intelligence, etc.

Of course if you statistically sample from that in convincing ways it will convince you it has the properties of the systems (ie., people) which created its training data.

But on careful inspection, it seems obvious it doesnt.

Bugs bunny is funny because the writing staff were funny; bugs himself doesnt exist.

If you “sample” this enough to be reasoning in a general manner, what is exactly the problem here?

Magic “reasoning fairy dust” missing from the formula? I get the argument and I think I agree. See Dreyfus and things like “the world is the model”.

Thing is, the world could contain all intelligent patterns and we are just picking up on them. Composing them instead of creating them. This makes us automatons like AI, but who cares if the end result is the same?

The distribution to sample from mostly doesn't exist.

Data is produced by intelligent agents, it isn't just "out there to be sampled from". That would mean all future questions already have their answers in some training data: they do not.

See for example this exact tweet: pre-2021 coding challenges are excellent, post-2021 are poor. Why? Because post-2021 didnt exist to sample from when the system was built.

At the minimum, chatgpt displays a remarkable ability to maintain a consistent speech throughout a long and complex conversation with a user, taking into account all the internal implicit references.

this to me is the proof it is able to correctly infer meaning, and is clearly a sign of intelligence. (something a drunk human has trouble doing, for example).

"I have seen the output and it matches what I consider to be conversation"

Well yeah, it's been trained to produce output that would look like conversation.

it's not what i meant : you can have a full conversation and then at some point use "it" or "him" , and based on the rest of the sentence, it will understand what previous element of the conversation you were mentionning..

This requires at least "some" conceptualisation of the things you're talking about. It's not just statistics.

It does not require conceptualization, pretty sure the "understanding" of previous references comes from this: https://arxiv.org/abs/1706.03762

This is exactly statistics.

> Bugs bunny is funny because the writing staff were funny; bugs himself doesnt exist.

Excellent analogy, and I appreciate analogies (perhaps even a bit too much). Will be using this one. Thank you!

> As for your analogy, I'm not sure we know enough about human intelligence core mechanisms to be able to dismiss NN as being fundamentally incapable of it.

If there's one field of expertise I trust programmers to not have a clue about it's how human intelligence works.

What makes you so sure we are capable of it? Gut feeling? How do you reason, exactly? This answer is worth billions so why not enlighten us.

You don’t know? But you feel you have “it”, this magical substance, this substrate of reason itself? Reasoning is build out of what, exactly?

Sorry to be that guy, but I fail to see more than word play.

What makes you so sure diluting things doesn't make them stronger? I mean, you don't know any physics, chemistry or biology -- but it's just word play right?

I mean, there isnt anything called science we might used to study stuff. You can't actually study any intelligent things empirically: what would you study? Like animals, and people and things? That would be mad. No no, it's all just word play.

And you know it's wordplay because you've taken the time to study the philosophy of mind, cognitive science, empirical psychology, neuroscience, biology, zoology and anthropology.

And you've really come to a solid conclusion here: yes, of course, the latest trinket from silicon valley really is all we need to know about intelligence.

That's how the scientific method works, right?

Sillicon Valley releases a gimmik and we print that in Nature and all go home. It turns out what Kant was missing was some VC funding -- no need to write the critique of pure reason.

>What makes you so sure diluting things doesn't make them stronger

Alcohol diluted with around 30% water makes it 'stronger' at killing bacteria...

I mean it's easy to say "just science that shit", and then forget we've been spending decades and billions of dollars doing just that.

Let’s all see and be amazed by the absolutely breathtaking achievements of those fields in the domain of AI…

The thing is that we've known how to do reasoning with computers since the 1960's at least. Here:

https://dl.acm.org/doi/10.1145/321250.321253

That's the paper introducing the Resolution principle, which is a sound and complete system of deductive inference, with a single inference rule simple enough that a computer can run it.

The paper is from 1965. AI research had reasoning down pat since the 1970's at least. Modern systems have made progress in modelling and prediction, but lost the ability to reason in the progress.

Yeah, we totally "scienced that shit" as you say in a comment below. And then there was an AI winter and we threw the science out because there wasn't funding for it. And now we got language models that can't do reasoning because all the funding comes from big tech corps that don't give a shit about sciencing anything but their bottom line.

> What makes you so sure we are capable of it? Gut feeling? How do you reason, exactly?

It never fails: When faced with the reality of what the program is your average tech bro will immediately fall back to trying to play their hand at being a neuroscientist, psychologist, and philosopher all at once.

You did not answer a single thing and maybe I am those things. You don’t know me.

Are you any of those things?

>> Honest question: Why so many people attribute "thinking", "knowing, "understanding", "reasoning", "extrapolating" and even "symbolic reasoning" to the outputs of the advanced token-based probabilistic sequence generators, also known as LLMs?

It's very confusing when you come up with some idiosyncratic expression like "advanced token-based probabilistic sequence generators" and then hold it up as if it is a commonly accepted term. The easiest thing for anyone to do is to ignore your comment as coming from someone who has no idea what a large language model is and is just making it up in their mind to find something to argue with.

Why not just talk about "LLMs"? Everybody knows what you're talking about then. Of course I can see that you have tied your "definition" of LLMs very tightly to your assumption that they can't do reasoning etc., so your question wouldn't be easy to ask unless you started from that assumption in the first place.

Which makes it a pointless question to ask, if you've answered it already.

The extravagant hype about LLMs needs to be criticised, but coming up with fanciful descriptions of their function and attacking those fanciful descriptions as if they were the real thing, is not going to be at all impactful.

Seriously, let's try to keep the noise down in this debate we're having on HN. Can't hear myself think around here anymore.

Just curious why you’d respond so much to him and also add nothing to the discussion?

Not him but I am also extremely frustrated by the fact it is impossible to have a real discussion about this topic, especially on HN. Everyone just talks past each other and I get the feeling that a majority of the disagreement is basically about definitions, but since no one defines terms it is hard to tell.

Hang on, how is it fair to ask me why I "add nothing to the discussion" when all your comment does is ask me why I add nothing to the discussion? Is your comment adding something to the discussion?

I think it makes perfect sense to discuss how we discuss, and even try to steer the conversation to more productive directions. I bet that's part of why we have downvote buttons and flag controls. And I prefer to leave a comment than to downvote without explanation, although it gets hard when the conversation grows as large as this one.

Also, can I please separately bitch about how everyone around here assumes that everyone around here is a "he"? I don't see how you can make that guess from the user's name ("drbig"). And then the other user below seems to assume I'm a "him" also, despite my username (YeGoblynQueenne? I guess I could be a "queen" in the queer sense...). Way to go to turn this place into a monoculture, really.

I don't think there's anything inherently different algorithmically or conceptually.

Our brain is just billions of neurons and trillions of connections, with millions of years of evolution making certain structural components of our network look a certain way. The scale makes it impossible to replicate.

What do you mean 'impossible to replicate'. With current technology, or in general?

Possibly both? Certainly the first of the two.

it kind of does “understand” when humans supervise it during training and they are able somehow relate and give mostly coherent responses. It may not be feeling it but it does seem to “understand” a subject more than a few people

But we do not even know whether GPT-4 is 'just a LLM'. Given the latest addons and the fact it can do some mathematics, I think there is more under the hood. Maybe it can query some reasoning engine.

This is why I think it is so important for OpenAI be more open about the architecture, so we can understand the weaknesses.

Reply on news.ycombinator.com