Hacker News Reader

Comment by f6v | Hacker News Reader

> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.

I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?

I'm especially confused about a prior statement as a scientist:

> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).

How does a set of beliefs help with reasoning?

> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.

We also often do the same as humans.

View on HN

Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.

Consider a calculator, a machine capable of valid transformations wrt the semantics of arithmetic.

Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.

An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.

However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.

I’m not so sure. While I completely agree on pure transformation, I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.

The interplay between deductive frameworks and inductive frameworks is still always buoyed by the black swan problem… which is a the heart of Hume’s solipsism. Kant’s pure reason never considered instinctual logical gates that come prepackaged in the mind. That seems frighteningly close to a prediction algorithm, no matter how much we wrap it in a concept of understanding. In the end, we’re just building a model with a minimal error rate.

Western tradition has firmly put pure reasoning in a cage. It serves its purpose only as a source of hypotheses, conjecture and thought experiments. Abrahamic religions, while still not accepted sensu proprio by many, are much less mystical than the Eastern Religions. You can see the difference in the unusual creative process of Ramanujan.

Which tradition are we to follow then whilst training the AI models? Pushed beyond their limits today's models tend to hallucinate and cannot be accepted sensu proprio. Philosophers will call it speculative reasoning. And yet, some speculation can be useful in breaking new grounds. Anyway, I have no answers. Just musings.

Philosophy isn’t “western,” it’s reason. The “non-western” traditions you reference are effectively religious and are not based in rationality.

> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.

I think it’s not a question of it existing, but to what extent does conscious can control the organism (as opposed to being a passenger). There’s some intriguing research on the topic.

> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.

Based on what? Just because you can model the weather using math doesn't imply at all that the weather is the product of a mathematical model itself. I don't see how reasoning is different.

We deliberately construct LLMs so that they approximate certain relations typical of reasoning. Suppose they become extraordinarily successful at doing so. What exactly have we demonstrated? That these effects can be artificially reproduced through such a mechanism. But how would it follow that the thing being reproduced must itself be nothing more than that mechanism?

> Based on what?

Based on the face that we have zero explanation for consciousness. Based on the face that biologically neural networks are exactly a kind of neural network.

My point isn’t that we’re able to explain consciousness as an illusion. It’s that we’re able to explain consciousness-like behavior presumably without consciousness, and the infrastructure look uncomfortably like we do.

What does consciousness have to do with reasoning? You're conflating two different things.

Consciousness is the what-it's-like of sensation you have as an organism from your sensory modalities and having a body. For human, that's colors, sounds, tastes, smells, feels, feeling of balance and what not that happen in perception, hallucination, memory, imagination, inner dialog and dreams along with emotions. Each sentient organism has their own sensations and emotions making up subjective experiences that nobody else has access to except inference from behavior and language (for humans).

That's not reasoning (just the having sensations/emotions). Reasoning is an adjacent activity that can be conscious but also unconscious. And it's hard to say that colors, sounds, emotions are illusions. What does that mean? We do experience them. There is something it's like to see color which is different from what it's like to feel pain or to be in love. And none of those (presumably) are what it's like when a bat experiences echolocation (there are many examples from the animal kingdom if one takes issue with echolocation).

>Consciousness is the what-it's-like of sensation you have as an organism

>Each sentient organism has their own sensations

This does not follow besides feeling like it should.

You made me start listening to Critique of Pure Reason [1] :-)

1. https://www.youtube.com/watch?v=cyhXtJN2AQQ

> Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.

Yes, it reasons through machine instructions in the silicone.

Can someone explain how we reason that 2+2 is 4.

Suppose you put an LLM in a "inference -> observe -> retrain weights" loop.

Supervised training uses this precise loop. The difference between inference and observation is called the loss function.

Models are now starting to show emergent behavior that they were not explicitly trained for. This is leading to the current debate.

Well, yeah, I understand there're all kinds of theories about this. That wasn't my question though.

Sorry, I misinterpreted your question:

Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning?

As:

Do we have evidence humans have an independent set of beliefs not shaped by observation and experience?

Because, that will be a priori or pure reason.

and some ... experience, actual lived experience.

> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.

I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.

For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.

> How does a set of beliefs help with reasoning?

You cannot have a syllogism without propositions.

> I think an analogy would be helpful.

It’s not, because it’s not evidence for the argument in the article.

> As a human, you WOULD know the idea of man

You’re making a claim about the fundamental mechanics of human biology. I don’t take these at face value. For one, language (in a very broad sense) put a major selective pressure on our species. So, to test your hypothesis, we’d need to find people who do not use any language and somehow test they have an abstract idea of a man.

To put it otherwise, I don’t know whether our reasoning can be decoupled from speech or language. There’re some bits and pieces here and there, like deaf people being at higher risk of neurodegeneration.

And let’s not forget there’re claims that AI models develop internal representations of various concepts. Anthropic made a very interesting claim, but I’m not an ML expert so can’t judge whether it’s sound.

Because he's writing about epistemologies: Sets of rules for reasoning over beliefs.

In such systems, a belief is a proposition you accept as true. (provisionally at least)

For example as a scientist, you probably use the epistemology called empiricism all day. In empiricism, beliefs are justified by observation.

> Because he's writing about epistemologies: Sets of rules for reasoning over beliefs.

He’s making a bunch of statements without providing evidence. I’d be insanely grateful if someone who’s in that field could provide some here.

In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.

How can you know that, when every human who reasons knows a lot? We spend years in a pre-rational state, and by the time we can reason reliably we have ingested a huge amount of evidence about the world. Memories are just a snippet of what you learn by observation; you don't have a memory of gravity but you do have an internalised model of gravity derived from what you learnt as a baby.

I don’t know why you guys keep rehashing what the author said. I don’t really have a reading comprehension problem. I have a problem with someone taking it a face value. See above for my other responses.

Yea.

I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.

They do not ask whether or not they dream of electric sheep. Unless prompted to do so.

That's part of human reasoning, being prompted, but so is independent thought.

So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.

In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.

This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.

If somehow independent reasoning then we're getting very close to something alive that would need rights.

In my view.

And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.

> And I have yet to have an experience with an LLM...

I hope you agree that this question is too important to leave to gut feelings.

Humans have a poor track record in that regard.

No question. As we were discussing it needs measured somehow.

Right. So I'm wondering about the relevance of you citing your gut feelings. It seems like you are putting stock in them. No?

> they are incapable [...] of independently starting a reasoning task.

This is a simple and intentional design choice though. Biological lifeforms are always on and always receiving sensory input. LLMs don't functionally exist out side of when we decide to run them. An always on agent with a looping prompt of "if you aren't doing anything else, ruminate" overcomes this limitation.

Not really. Noone has to prompt you to ruminate.

In fact, if your brain were removed from your body, and you were locked in a room that was completely empty and precisely calibrated to your brain's ambient temperature, and you were then ordered on your life to not ruminate. Well -- I would posit that you wouldn't last very long.

There's something else going on with human cognition -- and reasoning by extension -- that LLMs are, to date, not replicating.

With the big caveat of AFAIK. I'm not in one of the labs close to this stuff.

> Not really. Noone has to prompt you to ruminate.

Why do the internal mechanics need to mirror how it works in humans?

I have agents that receive sensory input (video/audio) which run continuously processing it, receiving events as they occur. When something strange happens they notice and take action (i.e. there's an unrecognized person in the living room -> ask who they are.)

Presumably you've seen the agents that play video games - build things to progress in Minecraft, plan routes, avoid adversaries, etc.

Why doesn't that count?

I feel like you're failing to consider anything outside the chatbot UI.

EDIT: The first morning my agents had access to an outside video stream they commented on the sunrise. While I was sleeping through it.

You make a lot of out of context assumptions and feel.

I'm sorry? I'm answering your claim:

> they are incapable ... of independently starting a reasoning task.

I have provided several examples of them independently starting reasoning tasks.

... and I'm wondering why you seem to be entirely unaware of such examples. The most likely explanation being you are only aware of the chat interface.

... and I have absolutely no idea of what you mean by "... and feel".

I think you’re on the right path. My first thought was whether lack of constant sensory input is an issue. We’re always feeling something (save for sensory deprivation chambers, but even then…)

Or put it other way, lack of feedback loops between the “brain” and the environment.

Reply on news.ycombinator.com