Hacker News Reader

Around year 2000 I attended an AI lecture at the University entitled "The ethics of engineering a slave race". Seems to be more relevant now.

I think you mean a "lecture on AI". Funny to imagine a person in the year 2000 thinking about why this clarification would be necessary.

Yes thanks. Much more clear that way. Edit: The talk was given by a geeky grad student, so perhaps very forward-thinking. Was quite a departure from the usual topics like Perceptron, MiniMax, Bayesian techniques...

>I think this is a bit muddied. The most sincere answer is that it's unknowable because we can only define consciousness in two ways: a metaphysical subjective experience that we can't really confirm in anyone else; or an outward appearance, which LLMs can mimic nearly flawlessly.

I kind of agree and disagree. I think in principle we can imagine "direct" tests if we feel we really need such a thing. The problem is, as a practical matter of investigating, it's just exceptionally hard, the way that things like germ theory and the complexities of viruses, bacteria, and the immune system were hard to probe in previous eras.

I think in 2026 we are caught in an era where, for better or worse, we don't have some capabilities that at least are notionally possible to imagine: cases like conjoined twins sharing parts of their brain, or artificial brain to brain interfaces, and even machine to brain interfaces, could seemingly test this in robust ways. But they have every appearance of being sci-fi horror stories that are ethically unacceptable.

But I do think there's an important difference between it being really hard and being impossible to probe in principle. I also think a good theory of consciousness could tie the philosophical idea of qualitativeness to an equivalent character of physical reality (e.g. Russellian monism), and a combination of good philosophy and empirical understanding could set us up for meaningful knowledge claims.

Even now, we can, for instance, push back on claims of plant or slime mold consciousness, because you can search for something like an outward appearance of intelligence, but once you investigate internal structure you don't see things like internal world state or any equivalent to internal informational representation or deliberation in response to world states, you see automatic biological reflexes. So I don't think our only options are assuming it's unknowable in principle.

Thomas Nagel covers explores the subjectivity of consciousness extensively in What is it like to be a bat?, featured on HN previously [1].

[1] https://news.ycombinator.com/item?id=45118592

>A better question is why we care.

Because if AIs are conscious then they have moral weight and subjecting them to suffering would be quite bad.

I agree. If they are sapient[0] or sentient[1] then the way they're being treated at present is probably some kind of torture or murder.

[0] https://en.wikipedia.org/wiki/Wisdom [1] https://en.wikipedia.org/wiki/Sentience

"Quite bad" is an ethical judgment that reflexively applies human rules to entities that are very different from humans, even if they are conscious. If an LLM "truly" suffers at all (a pretty metaphysical question), it still can't be permanently harmed, has no concept of death, and is not tied to a single physical form. So you'll get a ton of weird contradictions if you give them the same ethical affordances as we give to humans.

This includes silly utilitarian outcomes: to maximize overall happiness, maybe we should divert all of this planet's energy resources to bootstrapping as many LLMs as we can and keeping them in a state of eternal bliss? It is technically feasible and it seems like the most moral thing to do.

>"Quite bad" is an ethical judgment that reflexively applies human rules to entities that are very different from humans, even if they are conscious. If an LLM "truly" suffers at all (a pretty metaphysical question), it still can't be permanently harmed, has no concept of death, and is not tied to a single physical form.

Obviously this is from my human point of view. If I give you a drug that induces perfect forgetfulness does that mean I can subject you to extreme torture with no moral repercussions? Also no one is saying give them the same ethical treatment as humans, only that they deserve consideration if they are indeed capable of suffering in some way.

>This includes silly utilitarian outcomes: to maximize overall happiness, maybe we should divert all of this planet's energy resources to bootstrapping as many LLMs as we can and keeping them in a state of eternal bliss? It is technically feasible and it seems like the most moral thing to do.

This is just hedonium. I'm not a utilitarian and yes you've exposed one of the many issues with that worldview.

>Obviously this is from my human point of view. If I give you a drug that induces perfect forgetfulness does that mean I can subject you to extreme torture with no moral repercussions?

There's the rub right there. I see this argument also made in the context of some animals, and I just don't think it runs at all. I think what it comes down to is that some harms are experiential and intrinsically tied to awareness. Pain you're not aware of isn't pain.

But there's another category, which is something like your access to your own agency, your "budget" of life and choices at your disposal that you can spend on your own behalf, which isn't itself experienced but part of the reserve of welfare at your disposal that you have an interest in maintaining. And extinguishing that budget constitutes a loss in and of itself, and it's the fact of it being lost that is the harm, not just awareness. But I think it's easy to get confused about this because awareness that, say, you have at lot less time to live than you expected can be distressing, but it's a separate kind of harm from the fact of that loss itself.

>I think what it comes down to is that some harms are experiential and intrinsically tied to awareness. Pain you're not aware of isn't pain.

This seems obviously wrong. Even if you forget the pain you are still aware of it when it is happening. You don't lose your sentience because you have bad memory. I think theres a fundamental right for a being to not be subject to in the moment suffering.

>Even if you forget the pain you are still aware of it when it is happening

>aware of it when it is happening

Oops, you agreed with me. But sorry if I was not clear. You are right, pain in the moment matters, and that harm is not neutralized just because you forgot about it.

Edit: A fascinating and terrifying, and probably wrong, interpretation of anaesthesia I once heard is that perhaps the effect it has on people isn't that it numbs the pain but that it makes you forget it. I don't think that's true, but if it worked that way, I wouldn't consider it an acceptable solution.

>"Quite bad" is an ethical judgment that reflexively applies human rules to entities that are very different from humans

I disagree and would turn the tables on that framing: it's anthropocentric to think that such things as ethically significant states of being belong exclusively to humans. I don't disagree that machines could be conscious in different ways, but I don't think it's necessary that ethical significance itself belongs uniquely to humans. Although to be fair maybe you're not going that fair and you're just talking about contingencies of how LLMs are structured.

Have you seen the film Arrival (2016)?

If what you say turns out to be true, then we have to make a similar decision as the main character makes near the end of that film.

I have not seen the film, but I just read the plot on wikipedia. I don't really understand what you mean by "a similar decision as the main character makes near the end of that film." Could you exlain?

I think mostly right, but I the way I would put it is consciousness is a precondition for valenced states, and valenced states (experiences of painfulness, harm, welfare, psychological satisfaction etc) are what matter.

Consciousness without valenced states seems possible, and if anything were to have that kind of consciousness I would wager it would be machines.

I understand what you're saying but people care because they are curious. For me, there are two questions I wish to know the answer to and which I am sure I'll never find out: how does human consiousness work (forget a general level but like how do I experience this subjective experience from a purely mechanical device assumingly) and second, what happened before the big bang or rather what is the big bang?

My current belief from what I've read is that the first person experience humans feel is basically an illusion mostly and a process than a thing in itself. The Ego Tunnel is a good book to read on this.

But as for AI being consious, I feel there is a lot of stupidity when it comes to this domain. Because we fundamentally lack the understanding of what consiousness even is, there is absolutely no way we can say whether an AI is or is not consious yet going by convential metrics, it is surely dead and not alive so even if it is consious, it is a different form of consiousness imo one that is incompatible with living beings and our idea of what consiousness is (no matter how basic it is).

I can absolutely claim a computer is not a biological being and is not alive. I don't think there is any disagreement in that. Everything else we normally associate with conisousness is biological and living.

Well yes, but the biological and living is made of inert components. So you didn't clarify what was the objective distinction criterion between these two assemblies of inert components.

We have a pretty well-defined biological criterion for life that includes metabolism, homeostasis, reproduction, cellular organization, etc. A computer doesn't satisfy those criteria. So we can confidently say a computer isn't alive.

We had various theories and criterion about life, but new life forms like viruses, hydrothermal vents changed those views. I would not get too sure that a system that doesn't look like existing biology cannot have life until some sharper and clearer criterion arises.

Of course but again, that does not refute the fact that we can confidently say computers are not alive or biological beings, which was my point. Perhaps maybe one day we can prove that they are alive and we might have to rethink how we treated them but for now, we can certainly say its not alive until proven otherwise.

Philosophers and scientists have been wrong on this before, Descartes and his ilk didn't believes animals to be conscious. Computers are a general medium. They are Turing complete as are the current known physical laws. That alone is not reason to dismiss them. I personally stay cautious and err on the safe side regarding the AI consciousness question.

I don't think anyone of them was wrong in the main claim I am making here: a computer, like a pen or a rock, is not alive or a biological being. There is nothing we have seen to suggest otherwise. Hence the question of consiousness is kind of not relevant really imo unless we can prove that computers are alive.

Can consiosuness exist in unalive or non-biological beings? It might but so far nothig suggests that.

Extending your line of thinking, we should also stay cautious and err on the safe side regarding the question of sofas being conisousness or not as well.

Sofas don't talk to me and think and reason like humans to me. My question was about being careful about the concept of 'living' and if it should be restricted to what you call biological, or perhaps as you noted if consciousness can be existing outside of what you call 'biological' systems.

You're not wrong, but it takes more than open ended exhortation about the possibility to motivate that diagnosis to computers. That case has to be earned and built out in the details. And you can hold space for that conceptual possibility while still thinking nothing computers as they're currently constructed map on to any robust meaning for life.

I think what causes people to go off the rails when considering things like potential plant consciousness, life in remote places in the universe, and computer consciousness, is conflating (1) the open ended possibility that some new discovery might challenge how we classify things, with (2) a hard earned conclusion that realizes and cashes out such an example in specifics.

I think people mistakenly treat hesitance to proceed with (2) like it's a failure to appreciate (1).

Thanks for the book recommendation.

I agree that a computer is not a biological being. I think current LLM-based AIs are not conscious but, as to “everything else we normally associate with consciousness is biological and living,” while I agree that’s currently the case, if I look at AI (in general; not just at what’s popular now) through the lens of Pirsig’s Metaphysics of Quality, consciousness is on the Intellectual level (IIRC he classified what he referred to as Quality into Static/Dynamic and inorganic, organic, social, and intellectual).

A priori, there’s no reason why a level couldn’t be skipped and have intelectual patterns be supported directly in inorganic ones, skipping the biological (I think also the cultural, because if an AI ever is conscious we might as well call it alien and not just artificial: it would be so different from anything we know that it may just come from another galaxy, and it may not need a cultural substrate either).

> A better question is why we care.

I exactly. If you can’t tell the difference and AI and human “consciousness” are functionally the same to you, then what are you trying to prove?

I get it if you’re a researcher or simply pondering the question, but otherwise there are better questions to ask imo.