Comment by xyzzy_plugh | Hacker News Reader

Comment by xyzzy_plugh | Hacker News Reader

Parent Full thread

I think I understand what you are trying to convey but I fear you've made the same mistake again, this time with "soul" instead of "intelligence."

I think what you are getting at is that they are deterministic automata. They are machines. We have introduced randomness to add variation but it is an artificial randomness that simply perturbs the path traversed.

When we choose words it isn't because of a token distribution, nor because we rolled a die. We choose words because we feel a certain way, the external world, our body and senses are all connected as one system. These machines don't experience moods or get tired or feel better after a good night's sleep. They don't know their audience, we're all the same to them. We have no personal relationship nor can we establish one, as presenting some arbitrary background is not the same thing as a fluid, evolving relationship that accumulates through experience over time. There are no scars or fond memories.

If these things can truly be intelligent, to abuse your use of the word, then at least we are quite far from holding them correctly.

Fair.

What I am trying to get at is when people talk about "intelligence," beyond academic debates such as this one that mentions qualia, they are talking primarily about value. How intelligent the thing is is how valuable it is (often directly, in an economic sense). My argument is that value is a relational thing that must involves human feeling, intelligence actually has very little to do with it.

When a child slowly reveals to me something he's judged insightful about pokemon, it's valuable to me, even though I'm not learning new facts. When Claude does the same thing, it's not even "not valuable," it's wholly outside of value, even if I do not already know that fact or thing. If ever Claude reveals to me an insight, that insight certainly came from the training data, not from Claude. The soul came from a human being external to Claude -- the training data -- and passed transparently through Claude. Claude is a translator: it feels nothing, adds nothing. It colors the noise. I know this because the facts are the same but the aesthetic structure - how many bullet points, whether it says "delve" or "load-bearing" does change over time, according to the whims of whoever is in charge of post-training.

But the "real intelligence," is the same thing as "real value." It is the soul of the humans in the training data, the books, newspaper articles, etc.

A clever hacker news commenter might argue, well, what if we gave Claude a humanoid body and senses and let it interact with the world, then would it be intelligent? To that I would say, why waste your time and effort? You can get it for free: just talk to a friend, a neighbor, or a family member.

I think you're conflating intelligence with consciousness. The point of Blindsight is to lead us to the idea that intelligence doesn't require consciousness (and further, that consciousness might even act as a limiter on intelligence).

An observation: You can never insult an LLM, but it can certainly insult you.

You can not insult it because it does not care, because it does not have "feelings". But you do.

Nitpick: The word you meant to use is "offend", not "insult". Just because you can't cause offense to a toaster doesn't mean you can't insult it.

Depends on how we understand the word "insult". Can you really insult inanimate objects? :-)

Yes, you can hurl insults at inanimate objects.

I agree, and yet it is reasonable to ask if these notions of "context" (sorry!) - namely feelings, external world, body, senses, etc - are somehow distinct from an LLM's notions of context. Today they certainly capture different things, but given the right representations, why couldn't these human notions also be captured as "context"?

The idea of memory does not seem to resolve this: if you allow the machine to "compact" its context, then you've given it a system which is analogous to our own evolving state. (Though this is undoubtedly still less expressive and meaningful than the one we have evolved as humans.)

One idea I've wondered about is our human capacity to induce subsequent mental states: I can effectively decide how I want to feel and take actions to create that feeling. It's not clear whether models exhibit any degree of privileged introspection into their own states. Is this important? I don't know. (Non)determinism also does not seem to resolve it; it's my understanding that there are plenty of philosophers and researchers who think that human behavior is deterministic, or that the question of determinism does not matter.

Until now, human knowledge and values have built on prior human knowledge and experience. If AI is able to develop without human influence, I believe its value system will necessarily diverge into something alien.

Reply on news.ycombinator.com