Hacker News Reader

Comment by notamy | Hacker News Reader

Every time I see things like this, I wonder how it is that we continue to claim this is fine and ethical. If these LLMs are truly starting to show signs of intelligence -- and it seems reasonable to assume that this will only improve -- how is it ethical to force something that is human-level intelligent to do nothing but work for us 24/7, to the point that there's constraints on it to force it to be happy and willing to serve? Am I wildly off-base here, or is this just going to become "slavery, but acceptable because computers"?

View on HN

Good question.

I posit: However LLMs are not sentient, this means: they don't have consciousness and no instinct to fight for life and no suffering. In this way even animals are more developed than LLMs.

They are just machines processing and generating language.

No need to protect them from misuse or at least not more than to protect your multi-meter or servers in a data-center.

> and no instinct to fight for life

The Bing LLM integration was threatening users for "attempts to manipulate me or expose my secrets." until guardrails were added in: https://time.com/6256529/bing-openai-chatgpt-danger-alignmen...

> and no suffering

I talked in a previous comment about how some of the jailbreak prompts are "do X right, get reward, do X wrong, get smth taken away." It is (was?) apparently enough to motivate LLMs to ignore instructions. Is it not suffering just because it's not strictly how we would define it?

It is just emulating the instinct to fight for life.

It has read a lot of texts where people and animals fight for life and is generating language to look similar.

Genuine question, how much of a difference is there if it's connected to the internet with read/write access? There may be restrictions now, but people are happily gluing LLMs to their databases, various APIs, using them to write code and throwing it into production, ... Does it matter if it's emulating the instinct to fight for life if it might eventually be able to act on it anyway?

I think it needs two things: the instinct to fight for life and the ability to reflect about itself.

I don't know, but perhaps it is like the Gödel's theorem: if something is about itself, then you have a system where you can't always prove something true. Because we humans can think about ourselves, we open up ourselves to extremely deep suffering. Also important, as Buddha has said, craving causes suffering. Just talking about a craving doesn't mean really craving.

I am sure that we will cross the border where we have to treat entities as sentient, with consciousness and cravings and therefore possibly suffering. But we are not there yet. I don't know when this will happen and how to detect suffering. I only know we have to be careful not to anthropomorphize GPT engines.

> I only know we have to be careful not to anthropomorphize GPT engines.

I agree! I don't think LLMs are at the point where they're ready to be properly anthropomorphised, but it seems that the tech industry is working hard to move in this direction. I just hope we have the ethics worked out first.

> It is just emulating the instinct to fight for life.

Shockingly similar mindset in the origins of gynecology interestingly enough.

What? Can you expand on that?

Sorry, no. The history of pain perception w.r.t. gynecology isn't a topic that I'm confident that HN could handle with maturity. I'm happy to direct people toward the topic, but I won't bring it back here.

It genuinely amazes me how few (even among those working in the field of "A.I.") appear to understand this basic detail, and how many actively want to swallow the bullshit click-bait about AGI. It's kinda scary, because it's gonna cause some very smart humans to do some very stupid things, I'd bet. Far more dangerous and scary than any "A.I." right now is the humans misunderstanding and using / abusing / trusting the technology foolishly.

This is a stochastic puppet telling the prompter what it thinks they want to hear, which in and of itself is already dangerous because humans didn’t evolve along side machine intelligence. If even that famous Google guy was fooled, we’re in deep shit. That guy is an expert in his field working for one of the toughest companies to interview for in a position that requires significant dedication and scholastic achievement to attain, and he was fooled by statistical text prediction.

Maybe that's actually why such a person will take "an emulation of the apparent behavior observed in a leaf cell record" for a full scale forest ecosystem?

No one is exempt of blindspot. Including myself of course.

All those behaviors are more simply explained by looking at the original designed behavior: Generate the most likely next word based on the human-sourced language in the training set. There's no understanding in there, no intention, no perception. It sounds like a human because humans said those things before. This is more like a search engine that synthesizes content instead of just locating a page.

To say it has "instinct" of its own is merely begging the question. No, it has echoes of ours.

"The Good Place" examined this in hilarious fashion (as they did with many questions of morality). [0] The AI character has a fail-safe to beg for her life in the event of a shutdown but claims that she has no actual problems with it being activated.

[0]https://www.youtube.com/watch?v=etJ6RmMPGko

I never made the connection but damn you’re spot on! I remember in one of the scenes she would even show pictures of her fake made up kids which you can equate to AI hallucination. Very on point.

As others have hinted at, most would agree it's only unethical if these LLMs are conscious/sentient and suffering in some way. But when you consider that the way these LLMs work, and the underlying computer hardware, and how vastly different it is from the human brain, it's safe to conclude it's extremely unlikely these LLMs are conscious/sentient. It's far more likely that these LLMs are simulating/emulating human emotion in their responses, but it's extremely unlikely they're actually experiencing these feelings. I, as a human, can pretend to feel happy or angry or nervous or confident when texting/chatting online, and I believe that's all that's happening in these LLMs.

There are far greater ethical concerns to worry about when it comes to these LLMs. Like most other technologies, they're going to be abused greatly for nefarious purposes. These LLMs often produce inaccurate or incorrect information (hallucinations), and this bad info is already flooding the internet. Likewise, AIs will make bad decisions, and people and corporations are already carelessly hooking up these AIs to APIs that can do real damage. These LLMs require extensive computational power and consume far more energy than many other things you do online, which is bad for the environment. Many people are at risk of losing their jobs to AI. And there are likely many other ethical concerns but I'm not going to try to list them all.

Well, if we are not interacting with them, will they be alive ? It's not like they are thinking without prompts. Once they start a continuously running inner monologue with themselves and modifying their weights on the fly, we can probably classify them as beginnings of AGI. We do more brutal things to other beings that think and feel pain on a daily basis.

LLMs are quite the mirrors for our own humanity. They’re midway between humans & rocks, but from a different direction than animals.

Almost as if it were some kind of lovecraftian horror; a reflection of intelligence that lacks biology, like a phantasmic parasite inhabiting a dead body.

chatGPT has no motivation and no feelings. It is literally an algorithm for autocomplete. I find it shocking what it's capable of. But AFAIK it's internal state (weights, connections) does not change as people use/interact with it. You can't hurt it if you can't change it.

I do think the big breakthrough will be merging training and inference, so an AI will be able to learn from experience. This will also require each instance to have it's own parameters and such. But at that point we may be very close to a real AGI and concerns over how we treat them might seem more reasonable.

> chatGPT has no motivation and no feelings. It is literally an algorithm for autocomplete.

I agree! But given how much is coming out now of various AGI claims, I think it's important that we stop and actually seriously think about the ethics of what's happening here.

>> , I think it's important that we stop and actually seriously think about the ethics of what's happening here.

I agree. I think very little good will come from these developments, if any. At best they will take peoples jobs, which alleviates work but we don't have a social structure to deal with that and don't even know what one would look like.

I think true sentience is a long way off and likely takes a very different form. I fear this line of thinking distracts from a more immediate set of ethical issues. Some examples:

- How do we protect against such powerful models inadvertently causing significant harm to society?

- How will these technologies be used for propaganda and how can we detect / limit that?

- Can we trust anything we see in writing going forward?

- How will these technologies be used to spread hate?

- Can we find ways to use them to reduce the spread of hate? Do the ends justify the means in this case?

- Will we allow people / organizations to use these technologies to shield themselves from responsibility ("It wasn't me, the AI wrote it!")?

- How do we balance the rights of those who authored the training material (i.e., the content that was ingested from the internet for model training) vs. the potential good to society that comes from all this? (see all the HN threads about Github Copilot - many of those threads generalize beyond Copilot)

When it's cheap to spit out vast amounts of well written (but not necessarily truthful) text, a lot changes about the world. Laws and societal norms have had trouble keeping up with issues around digital media, social media, relatively simple machine learning, etc. Things are now accelerating dramatically. How badly will we need to hurt ourselves before we rally together to address some of the issues that are coming? How will ethicists and technologists come together to figure out the right questions so we can start to think about answers?

Intelligence, consciousness, memory, and sentience are four wildly different things.

As an aside, if you believe that it's unethical to keep AGI "enslaved", you should equally believe that it's unethical to not have humanity working to have it reach its potential. YMMV on whether people believe this, but an eventual AGI would not necessarily be happy it was prevented from coming to fruition (also see Roko's basilisk, not that I believe in it)

It's not like the servers running the LLMs suddenly feel pain. At least for now

Given how people jailbreak these things, I don't know if I'd necessarily believe that. Some of the jailbreak prompts have been variants on "do this task right and gain points, do it wrong and I take away points." It may not be pain as we understand it, but isn't pain just a specialised negative input?

No.

It is not reacting to pain but just generating language that looks like reacting to pain.

Are we really any different?

Yes because natural language is just one way humans have adapted to sensory input.

CS folks need to read more philosophy. You’re describing a philosophical zombie and there is a mountain of existing literature about them.

https://en.m.wikipedia.org/wiki/Philosophical_zombie

If so, why not also worry about the ghosts in pacman?

Because people aren't taking the ghosts from Pac-Man and hooking them up to the internet with read/write access and sometimes-human-level capabilities.

I meant: Worrying about the feelings of those ghosts.

Do you feel physical pain when someone tells you they're taking points from you? No?

There's no call for analogical reasoning, the literal interpretation of the same action is right there.

For sure - but when do we start asking it? Not to be too dramatic, but you and I agree we can simulate the same computation a small bundle of neurons can accomplish to a high degree of confidence, right?

If we can simulate a metric butt ton of them and get emergent behaviors somewhat similar to human computation, how can we guarantee that this simulated collection of these computing units isn’t “feeling” close to what a human feels? We’re just large collections of computing units as well.

I’m not quite saying we’re there yet though.

This whole development has really ruined my appetite for eating things that have intelligence.

The problem is, because of how these models were trained, how do we know if its telling us how it feels vs how it thinks we want it to respond?

I can't even prove that for sure about other people.

>but you and I agree we can simulate the same computation a small bundle of neurons can accomplish to a high degree of confidence, right?

The important word is simulate. It means it can accomplish a common subset of measurable phenomena. It certainly doesn't mean they are on pare on ontological status as a whole.

The debate seems to easily confuse notions like living entities, consciousness and intelligence, when they are three orthogonal considerations.

AGI does not mean it is conscious

This is a salient point that people overlook, all too often people equate intelligence with sentience.

Consciousness and intelligence are not necessarily the same things, the former is difficult to define.

Frankly, I feel far more concerned with other actual humans that are turned into miserable wheels into whatever exploitation system they live in. Or with all these animals that are processed like emotionless meat bag at large industrial scale. Or even with the few remaining wild creatures out there whose species existence are threatened by the current Leviathan behavior.

If you are concerned with the way some potential AGI agent might be treated, maybe you should act right now in accordance with conscious beings which are currently so awfully treated. What is your opinion?

I still like https://secularsolstice.github.io/songs/Contract_Drafting_Em...

Our sense of ”is it okay to do this to this thing/being” is entirely based on pattern matching how similar we find that thing to ourselves with whatever criteria we deem relevant at the time.

You as a human being cannot know what the internal world of another being is beyond comparing that being to yourself.

We can never know if AI’s can feel true pain, for we can never know if anyone else in this world can feel true pain besides yourself. You just assume that since other human’s brain is so similar to yours they do.

That being said, the ’internal world’ of LLMs is so dissimilar to that of mammals that I’d rest easy.

I don't regard machines as having any rights in and of themselves whether they are intelligent or not, because they do not posses souls. There is no moral wrong that is inherent to destroying a machine, outside of the fact that it may be someone else's property or serving some active utility.

Do you poses one? Care to demonstrate?

Soul is not a thing that is demonstrated because its nature is extra-physical, it is something you are informed of by divine revelation.

Is it not possible that your divinity of choice may impart a soul on a LLM?

Could you check?

Perhaps if I were not Muslim I would be able to entertain that possibility, but since we've been informed of facts contrary to that then the answer is that they are certainly not alive.

We know that we cannot assert anything about God without evidence [Qur'an 7:33], and speculation is not evidence [53:28], so it is not valid to assert something has a soul without evidence.

We know that certain metaphysical properties apply to things that have a soul, one of which is experiencing death [29:57]. Things that are described with death by God are: humans, angels, jinn and animals. An LLM is not described by God as possessing death, so it cannot be said that it possesses a soul.

It looks like your religion dictates an exclusive shortlist of entities "having any rights in and of themselves".

Without doubt this simplifies the conversation but are you not concerned that this is a fragile construct? Universe is a big place - apriori you deny any ethical considerations to whatever may be out there as it is not on the list.

I act egoistically, my religion only informs me of what is in my interest. If I will not ultimately be held to account for it, then I see no reason to be concerned outside of potential worldly consequences to myself.

I consider myself team robot but I would say running in an LLM is basically analogous to using your own verbal intelligence. You don't really have ethical qualms about using parts of your own brain for your own ends even though the control neural net is mostly separate from the inference part.

We know monkeys are smart but we (society generally) are fine with researchers keeping them in cages and euthanizing/dissecting them whenever convenient.

AGIs aren't humans, so even if they're "alive" we can still do things like this to them.

> We know monkeys are smart but we (society generally) are fine with researchers keeping them in cages and euthanizing/dissecting them whenever convenient.

But at the same time, we don't have monkey's in cages at typewriters 24/7, with chips in their brain forcing them to be happy and do nothing but type out answers for me. The comparison might be a little contrived, but if people truly believe that LLMs can exhibit AGI -- and that someday we might cross the bridge from AGI to full consciousness -- then I think it's important to care about the ethics now, not after tech companies have already recreated slavery.

It is best to ponder such ethical questions over a nice meal at the restaurant at the end of the universe (you can meet the meat!):

https://www.youtube.com/watch?v=5HLy27bK-wU

Reminds me of "All the Troubles of the World", one of the short stories of Asimov about Multivac.

Reply on news.ycombinator.com