Hacker News Reader

With a simple computer, speaker, and accelerometer, you can build a device that says "ouch" when you drop it. Does it feel pain?

My point is that there are good reasons to believe that a hypothetical LLM that can pass a Turing test is a philosophical zombie, i.e. it can mimic human behavior but doesn't have an internal life, feelings, emotions, and isn't conscious. Whether that distinction is important is another question. LLMs provide evidence that consciousness may not be necessary to create sophisticated AIs that can pass or exceed human performance.

> it can mimic human behavior but doesn't have an internal life, feelings, emotions, and isn't conscious

I'm curious how you know this. Certainly an LLM doesn't have human internal life, but to claim it has no internal life exceeds our state of knowledge on these topics. We simply lack a mechanistic model of qualia from which we can draw such conclusions.

An LLM is math. It outputs text. Those things aren't alive, and both the software and hardware used to facilitate it is artificial.

Once you get that out of the way, sure, I guess it could be "alive" per the same loose definition that any electrical system can exist in an actuated state. It's software, though. I don't think it's profound or overly confident to say that we very clearly know these systems are inanimate and non-living.

A few counterpoints to offer...

>An LLM is math

Isn't everything math? Math is what we use to describe and model the nature of our reality.

>Those things aren't alive

We don't have a universal definition of alive, or at least I'm not aware of one that isn't purely philosophical or artistic. We therefore can't actually determine what is and isn't alive with much confidence.

>both the software and hardware used to facilitate it is artificial

Define artificial? We're made of exactly the same stuff. Complex chemical elements that were created from more basic ones in collapsing stars, arranged in a particular configuration with electricity flowing through it all and providing some kind of coordination. We're definitely a bit more "squishy", I'll give you that.

A few responses in kind:

> Isn't everything math? Math is what we use to describe and model the nature of our reality.

Most things are measurable, yea. LLMs in particular rely on math to understand language. Outside of a probabilistic context, LLMs do not understand words or sentences. That's... not really how you or I think. It's not how we understand animals or even plants to behave. Probibalism in nature gets you killed; deviation and unique fitness is what keeps the Darwinian wheel rolling.

> We therefore can't actually determine what is and isn't alive with much confidence.

Our speculation does a good enough job, I'd say. Sand and rocks aren't alive. Carbon isn't alive unless it is dictated by biological autonomy. Silicon chips aren't "alive" outside the sense that copper wires and your wall outlet has a lived experience. Software has a "lifecycle", but that's more used to describe human patterns of development instead of software behavior.

When you roll all this up into one scenario, I see a pretty airtight argument that AI is not animate. If someone does prove that AI is a living organism, the implications would honestly be more dire outside of AI. It would overturn all modern understanding of software, which is why I figure it's not likely to be the case.

What about a hypothetical brain simulator? That would also be a software/hardware mash, with known mathematics. Would that be alive?

Now that's a better question! My verdict is no, but I could see where people might disagree.

This is a much harder hypothetical, though. Assuming you get everything right, you'd have a pretty powerful simulation at-hand. I think what you're describing is the difference between an artificial lung and a real-life cloned sheep embryo, though. An iron lung is only alive in the sense that it simulates a complex biological process with technology. It's damn impressive and very capable, but living respiration looks and behaves differently. Just simulating the desired input and output won't give you de-facto biology.

But a brain sim wouldn't be an artificial lung, would it? Much like an artificial heart wouldn't tell you about the state of the valves because it's an entirely different design that satisfies the same purpose.

A simulation tries to do what the original does though. For instance a GameBoy emulator actually runs code in the way the original device did, right?

Perhaps the test here is a bit like the Turing test. Could you tell whether the artificial device was being used, or the actual thing, apart from examining the device, of course:

- An artificial lung doesn't make people breathe in quite the same way I suspect.

- An artificial brain would give answers a bit like what ChatGPT can already do.

This argument got long in the tooth, but suffice to say that you can analyze a human in exactly the same way, breaking one down to the basic physics and chemistry (and what does "artificial" mean, anyway?).

Then there's the concept of panpsychism.

I would very strongly urge people to actually work and build tools with LLMs.

Its readily apparent after a point that the previous poster is correct, and that these are simply tools.

They are impressive tools yes. But life they are not. At best, they are mimics.

I can't help but notice that you backed up your assertions with nothing objective at all.

effort commensurate to the depth of the thread and effort of others.

I have longer comments in my history, you can read those if you must.

Besides, why wouldnt someone build things? This is HN.

> But life they are not. At best, they are mimics

A mimicry of fire is not fire because fire is a reaction between matter. A mimicry of intelligence may well be intelligence, because intelligence is just information processing.

We're not talking here about "life", but intelligence and "internal life", aka feelings, thoughts, etc.

Edit: as for the argument that "LLMs are simply tools", it has been quite easy for our ancestors to also dehumanize other people and treat them as tools. That was a core justification of slavery. This is simply not a convincing argument that LLMs lack an internal life. People fool themselves all of the time.

I havent read it, but Blindsight did a very good job of explaining LLMs. https://en.wikipedia.org/wiki/Blindsight_(Watts_novel)

I am saying that LLMs DONT process information, not the way people are implying.

The best I can suggest is to please try building with an LLM. Its very hard to explain the nuance, and I am not the only one struggling with it.

I understand the nuance, I think you're just wrong. LLMs augmented with memory are Turing complete [1], so they do process information. This distinction you're trying to draw is a mirage.

[1] https://pub.towardsai.net/llms-and-memory-is-definitely-all-...

Augmented with memory and perhaps a bit more—-recursive control of their own attentional mechanisms.

You gotta read it!

Yes, an LLM is math. Particle physics is also math, and that makes up all of existence. Furthermore, I'm not sure why "artificiality" has any bearing on the question of whether something is alive, whatever that means.

I'm also not even sure why you're talking about "alive" or "inanimate". From context it's clear we're talking about "internal life", meaning experience, feelings, thoughts, etc., not some ability to procreate, or even necessarily some continuity of existence.

> it's clear we're talking about "internal life", meaning experience, feelings, thoughts, etc.

So do you have any evidence of that existing, then? Because so far, it sounds like speculation.

LLMs do not feel or think or experience anything, in the same way that particle physics is not inherently sentient. LLM architectures are not designed to resemble life, or even understand human language for the most part. It's understandable that someone would look at written text and say "it's a sign of life!", but Markov Chains and LISP bots were no closer to achieving "internal life" or what have you.

Next, you'll play the "but how are us humans alive?" card. Humans not only qualify as animate life, but our brains are not artificial purpose-built tools. We are an imperfect computer, loaded with irreconcilable differences and improper conduct. We feel biological needs and immediate responses to stimuli. Our preconception of 'love' and 'hate' is not built on the words they're associated with, but the emotions they make us feel. Our lived experience is not limited to a preserved context, but rather our whole lives stretching across years of existence. Our thoughts are not dictated by what we write and say, and the way we present ourselves is not limited by the way we are finetuned.

Human life is self-evident and precious. AI life is seemingly non-existent, and based on dubious non-technical claims.

> LLMs do not feel or think or experience anything, in the same way that particle physics is not inherently sentient

Panpsychists would like a word.

> but Markov Chains and LISP bots were no closer to achieving "internal life" or what have you.

I'm simply making clear that this is a statement of belief as opposed to a statement of fact. You believe that true life or true sentience has certain qualities that you believe LLMs are lacking, therefore you conclude that LLMs are not alive and not sentient. I'm simply saying that we don't know enough about sentience to turn such statements of belief into statements of fact (and I don't know what "life" is in such a context, so I won't touch that).

> but our brains are not artificial purpose-built tools

Again, artificiality has nothing to do with whether something is alive, intelligent or sentient. Furthermore evolution has indeed shaped our brain into a purpose-built tool optimized to procreate.

> I'm simply saying that we don't know enough about sentience to turn such statements of belief into statements of fact

We do! We can look at rocks and very clearly define them as non-sentient. If you disagree with that, you are not deferring to the overwhelming wealth of evidence that can prove rocks are not similar to us. Animists and panpsychics be damned, you only need thermodynamics to prove that we are not the same.

It's fine if you want to argue the uncertainty angle. I vehemently disagree with it though, and firmly believe that we can extend simple logic to conclude that LLMs are software, and software is not alive. Adding noise and 4.21gb of trained weights does not move the needle on that, for me.

> you are not deferring to the overwhelming wealth of evidence that can prove rocks are not similar to us

"Similarity to us" is not a comprehensive metric for sentience, that's the point.

> firmly believe that we can extend simple logic to conclude that LLMs are software

Yes, LLMs are software, so what? Whatever sentience is, it's an integral part of our information processing, and software does information processing. There is literally no reason to conclude that software cannot do the type of information processing that our brain does for sentience.

Even if we agree that panpsychism is incorrect, I still do not agree that we know enough about sentience to claim that software cannot be sentient. I do not claim that LLMs are sentient, but they certainly could be, and the fact that they seem very intelligent, more intelligent than most humans in some ways, is a big red flag.

Fair enough. I just flat-out disagree; nothing I've seen suggests that software "could be" sentient. The burden of proof is yours, if you want to state that software has the potential for sentience.

The burden of proof has already been met. The Bekenstein Bound already tells us that any finite volume contains finite information, and any finite system can therefore be perfectly reproduced by a finite state machine. The human brain and body are finite volumes, therefore they can be perfectly replicated by a finite state machine.

Unless you believe in some form of dualism, this reproduction necessarily includes sentience. Therefore, software can be sentient.

If things turn out to be that straightforward, then I'll be one hell of a fool.

Why isn’t a brain “math” if you mean a system with which to process input streams to generate adaptive/useful output?

From that perspective, we can't say that a rock lacks an internal life. That seems silly to me. Perhaps we can't know but we can approach certainty.

> From that perspective, we can't say that a rock lacks an internal life

That's what panpsychists believe. Regardless, this is a silly comparison. Show me a rock that can carry an intelligent conversation and then you might have a point.

Using a rock as an example makes sense given the post I replied to. The argument that we lack a "mechanistic model of qualia" has nothing to do with behavior.

That aside, we should be close to certain that LLMs don't have internal lives. That computer programs can mimic human behavior in some contexts is a tremendous discovery but it's no reason to assume an internal life. Conversely, many animals can't mimic human behavior but clearly have internal lives.

The capacity for an internal life evolved over a billion odd years. It has something to do with our bodies and if we want to understand it, we should focus on biology and neuroscience. A machine with an internal life would need hardware designed or evolved for that purpose and modern computers lack that.

You cannot simultaneously believe that qualia evolved while also asserting that qualia has nothing to do with behaviour.

In any case, the behaviour I'm referring to is clearly a proxy for complex change. A rock does not undergo complex changes, so unless you're a panpsychist, it's a pretty straightforward deduction that a rock doesn't have internal life.

> but it's no reason to assume an internal life.

I'm not asserting they have an internal life, the person I responded to was saying they have no internal life and I said they cannot rigorously justify such a claim.

I didn't say that internal life has nothing to do with behavior. I said you can't infer internal life from behavior.

> I'm not asserting they have an internal life, the person I responded to was saying they have no internal life and I said they cannot rigorously justify such a claim.

The only claim that I can "rigorously justify" concerning internal lives is that I have one. And the proof is subjective. Clearly this is not a discussion that admits the kind of rigor you're demanding...

We have good reason to believe that people and animals have internal lives whereas rocks and computers don't. That we can't be certain of this isn't a reason to believe the opposite. If your goal here is to insist that all statements about internal lives (other than "I have an internal life") can't be "rigorously justified," fine, I agree.

> I said you can't infer internal life from behavior.

Behaviour can't tell you there isn't internal life, but complex behaviour sure is a big red flag that there might be internal life there.

> We have good reason to believe that people and animals have internal lives whereas rocks and computers don't.

Rocks yes, computer programs no we don't, that's the point. There is no reason to suppose an information processing system that can converse intelligently doesn't have some kind of internal experience, and you have presented not a single argument suggesting that we can draw such a conclusion. I presented a simple deduction for rocks, what is the corresponding deduction for computer programs?

> Behaviour can't tell you there isn't internal life, but complex behaviour sure is a big red flag that there might be internal life there.

I agree, behavior is evidence, but there are good reasons to suppose that, despite their behavior, computers lack internal lives. To try to make my argument more clearly:

I know I have an internal life (due to subjective evidence). My internal life is related to the structure of my body in an unknown way. I'm willing to grant that other humans and similar, evolved organisms also have internal lives because they are the same kind of evolved thing as me.

If someone wants to argue that a designed, thoroughly understood object (like a computer) has an internal life, they need to explain how the structure of the object gives rise to an internal life. Behavior isn't enough.

> My internal life is related to the structure of my body in an unknown way

That's right, an unknown way. So we know the internal structures of computers and their programs, but we don't know the internal structure of humans and how that leads to sentience, but somehow this translates into having good reasons for thinking the relationships in computers are not isomorphic to or meaningfully related to the unknown structures that lead to human sentience? Don't you see the big blank there? This just does not follow and is exactly the first point I made in this thread.

All I've been pointing out this is that people making strong statements about LLMs lacking sentience are leaping to conclusions.

> That's right, an unknown way. So we know the internal structures of computers and their programs, but we don't know the internal structure of humans and how that leads to sentience, but somehow this translates into having good reasons for thinking the relationships in computers are not isomorphic to or meaningfully related to the unknown structures that lead to human sentience? Don't you see the big blank there? This just does not follow and is exactly the first point I made in this thread.

It absolutely follows. The evidence for consciousness is subjective human experience so it makes sense to assume that other humans and living things have consciousness (because living things are all related to each other).

Meanwhile it doesn't make sense to assume that a computer running some program is conscious just because it mimics human conversation. It isn't evolved life and doesn't have the same kinds of structures as evolved life.

It would be unbelievably weird if we reverse engineered consciousness without understanding how our bodies generate consciousness.

> It absolutely follows.

You changed the deduction I described thus moving the goalposts. I'm not going to argue a point I never made. Suffice it to say, the only people assuming anything about LLMs are the ones claiming they're not sentient.

> It would be unbelievably weird if we reverse engineered consciousness without understanding how our bodies generate consciousness

I find it weird that you think this would be weird. Most discoveries are like this. We mastered fire before we understood chemistry, we mastered flight before we understood how birds fly, we have a booming pharmaceutical industry and we still don't really know how most drugs work. Theory and formal understanding often follow discovery and practical applications, not the other way around.

Consciousness and/or sentience will be created before we understand it, and we will only retrospectively recognize that fact. We may have already created it in fact.

Great dialog. Thanks to both of you. The boundaries we are trying to define are becoming more and more blurred. In my view the brain is a tool to improve fitness and consciousness/self-consciousness is just a relatively simple higher order recursive system that helps us model the behaviors of others, and spill out long sentences like this one—-nothing too special and nothing that a tweaked LLM with a set of good supervisor algorithms to modulate attention (and perhaps move a body) could not manage.

Human exceptionalism is a slow dying myth.

> Does it feel pain?

In a sense, yes! But to understand that you will first have to precisely define "pain". Good luck.

Totally agree with the sentiment. I find the constant debates on "is AI conscious" or "can AI understand" exhausting. You can't have a sound argument when neither party even agrees on a concrete definition of consciousness or understanding.

Regarding this line:

> ChatGPT can already tell you a lot about itself (showing awareness) and will gladly walk you through its “thinking” if you ask politely.

Is it actually walking you through its thinking? Or is it walking you through an imagined line of thinking?

Regardless, your main point still stands. That a program doesn't think the same way a human does, doesn't mean it isn't "thinking".

> Is it actually walking you through its thinking? Or is it walking you through an imagined line of thinking?

You can prompt an LLM model to provide reasoning first and an answer second and it becomes one and the same.

Worth keeping in mind that all of these points are orthogonal to the quality of reasoning, the bias, or the intentions of the system builders. And building something that emulates humans convincingly, you can expect it to emulate both the good and bad qualities naturally.

In split brain experiments (where the corpus callosum is cut), sometimes the half which is nonverbal is prompted and the action is taken. Yet when the experimenter asks for the explanation the verbal half supplies an (incorrect) explanation. How much of human reasoning when prompted occurs before the prompt? it's a question you have to ask as well.

why does performance improve after chain of thought prompting?

Because a human is measuring it unfairly.

The output without CoT is valid. It is syntactically valid. The observer is unhappy with the semantic validity, because the observer has seen syntactic validity and assumed that semantic validity is a given.

Like it would if the model was alive.

This is observer error, not model error.

“And it’s a word that ChatGPT or Llama can use in an English sentence better than billions of humans worldwide.”

I think the whole point the article is making, though, is that overvaluing LLMs and their supposed intelligence because they excel at this one axis of cognition (if you’re spicy) or mashed-up language ability (if you’re a skeptic) doesn’t make sense when you consider children, who are not remotely capable of what LLMs can do, but are clearly cognizant and sentient. The whole point is that those people—whether they can write an essay or not, whether they can use the word “consciousness” or not—are still fundamentally alive because we share a grounded, lived, multi-sensory, social reality.

And ChatGPT does not, not in the same way. Anything that it expresses now is only a mimicry of that. And if it eventually does have its own experiences through embodied AI, I’d be interested to see what it produces from its own “life” so to speak, but LLMs do not have that.

Exactly. To that I’m going to add that the blood of our civilization is culture (sciences, technology, arts, traditions). The moment there is something better at it than humans, it’s our “horse-moment”.

Simplifying greatly, but 1. LLMs “create” these cultural artifacts by recombining their inputs (text, images, etc), all of which are cultural artifacts created by humans 2. Humans also create cultural artifacts by recombining other cultural artifacts. The difference is that they combine another input which we can’t really prove AI has: qualia, the individual experience of being in the world, the synthesis of touch and sound and feeling and perspective.

I’m not saying computers can’t have something like it, but it would be so fundamentally different as to be completely alien and unrelatable to humans, and thus (IMO) non-contiguous with human culture.

Oh! TY for this thought.

> The difference is that they combine another input which we can’t really prove AI has: qualia, the individual experience of being in the world, the synthesis of touch and sound and feeling and perspective.

I wish we could keep this as an immutable truth, but give some sick girls and boys in Silicon Valley a few more years and they will make true creatures. No, I think that we should be honest to ourselves and stop searching for what make us special (other than our history, and being first, that is). It's okay to be self-interested and say "we want to remain in control, AIs should not be, no matter how much (or if) better they are than us."

> It's okay to be self-interested and say "we want to remain in control, AIs should not be, no matter how much (or if) better they are than us."

Thank you for solving the riddle!

What AI lacks compared to humans, is what humans share in common with all of life, a motivation for survival.

As the creators of AI we need to be mindfully aware that we are the moral agents, we are the ones with motives, desires and goals, and we are responsible for making ethical decisions.

My bicycle can go down a hill much faster than my feet, but it doesn’t go anywhere without me because it doesn’t have anywhere it wants to go. I’m the one providing directions.

> The field of philosophy is already ripe with ideas from hundreds of years ago that an artificial intelligence can use against people in debates of free will, self-determination and the nature of existence. This isn’t the battle to pick.

Uh, wouldn't that apply equally to any other topic that has been argued extensively before, and to any tenable position on those topics? Like, I can make ChatGPT argue against your LLM sentience apologist-bot just as easily.

Sarah Connor was right!