An Ethical AI Never Says “I”
livepaola.substack.com
livepaola.substack.com
Worse - fragile world here - wait until humanoid robots become as cheap as drones and everyone can buy them. How many irresponsible drone pilots and drivers do you see? Just wait until they can have humanoid bodies walking around at their disposal.
We might have a lot of accidents due to sheer stupidity rather than malice.
AI wielding handguns is nothing , wait until they build skynet and invent time travel. Might as well give up now
The loud attention seekers do. They're not a majority unless you measure and determine it to be the case. I strongly suspect they're not.
My favorite way of illustrating this absence of self is to consider that in any conversation with an LLM, measures are always taken to ensure it doesn't happily continue both sides of the conversation, impersonating you and the "agent" in turns. Because that's what LLMs do - they predict what comes next for any given text. Individuals contained in this prediction (such as the "agent") never have any special importance, or selfness, to the LLM.
This is at least exemplified here: https://help.openai.com/en/articles/5072263-how-do-i-use-sto...
Notice how in the chat example, the output "Human:" is used as a stop sequence to make sure you don't get impersonated by the model.
They got the point, but they structured the narrative wrong. Sound familiar?
The reality is that an LLM should answer in the first person because it isn't answering at all! It's presenting a continuation of implicitly modeled text. A string of text containing an "answer" is a valid continuation.
The problem here is contextual identity. Calling it "AI" personifies it. Calling it a "language model" implies some sort of constructive parsing. An LLM is neither.
We should stop calling them "Large Language Models", too. Instead, we should call them "Text Inference Models".
"Large" implies that the size is a core feature, but really that is an implementation detail: smaller training datasets are less interesting, not a different category of thing.
"Language" implies that the patterns being modeled are specific to language. This is not true: GPT famously demonstrated the ability to infer an Othello game board structure, despite that structure not being defined with language. The thing being modeled is text. That's way more interesting, and has very different implications. The main implication people miss is that "limitations" are really just "features". For better or worse, a lie is structured with the very same patterns as truth. The more interesting implication is that a person writing implicitly encodes more than they intend to: including the surrounding circumstances that determine what text the person chooses to write, and what text they leave unwritten.
“Predictive Text 2.0” seems about right, and also has the benefit of mapping onto a technology that most people are already familiar with
As complex as they can be, every continuation is wholly dependent on the text in the training corpus and the text in the prompt.
But I think the biggest ethical dilemma in AI research is not going to be like everyone seems to think (that AI risks harming humans)-- Although this is almost certainly true (even to a large degree), I contend that the bigger risk might be the other way around. [1]
I think 100 or 200 years from now, future people will look back upon us as monsters: I believe the greatest ethical risk is that humans will abuse AI and cause it to experience true suffering.
I don't think we're ready to talk about this, because we're so certain that we possess some magical faculty that makes us different. And we continue to think that today's AI models are incapable of being conscious. But these words are very imprecise and we've always wielded them in very self-serving ways. I fear we may have greater capabilities than we think we do, at least insofar as we're not missing any fundamental ingredients to create brain-like AIs: from here, it's just a matter of trial and error.
How will we treat our creations? Well, human History doesn't bode well for any creature that doesn't look the same as us and that doesn't come from the same place as us.
Will we grant AI legal rights? I doubt it, at least not for a very long time.
---
[1] I think there's at least a small chance that we don't destroy ourselves with AI, vs there's pretty much zero chance that we're going to treat AI any better than we treat other creatures.
Yes, none of this makes any sense, but I got Asimov vibes, so more please.
I did expect to get very negative reactions when I wrote this comment (I know how people think)... But I do think it makes sense once you accept that our conscious experiences just emerge from information processing in the brain, and that there is nothing stopping such emergent behaviour from happening in computer systems. One type of machine can emulate another.
We already see incredible emergence at work in current AI, and we know we're not building AI with brain-like architecture, and we have tremendous knowledge of brain architecture. Once you put all these pieces together, it's just a matter of time before we create life-like intelligence, with much of its peculiarities, sensations, memories, cognitive biases and all.
"Believe" or "have faith" would be more accurate than "accept" here.
There's absolutely no clear evidence of the claim you're making, and plenty of other possibilities.
It's also an incredibly hand-wavy claim. Its information content is close to zero.
So there is still a missing process that turns organisms into self-sufficient survival bots and also gives brains some level of consciousness as a consequence and it isn't trivial to recreate. Otherwise we would be crawling in artificial minds with their self-induced purposes by now.
People have thrown vast amounts of neurons in a bag, and have tried organizing networks in highly regular ways, but we have yet to apply what we know about brain tissue organization. We've done the equivalent of throwing a bunch of wood and concrete into a pile, so it should be no surprise that it doesn't look like a home. IMO it's quite remarkable that this approach has taken us so far!
Do you think a Raspberry Pi, a 1970s electronic calculator, or a simple analog thermostat is conscious? They all do information processing. If they aren't conscious, what is it about the information processing that brains do that produces consciousness?
This is an obvious false dichotomy:
> There's no magic in the goo. It's all information processing
It doesn't have to be magic. Just to take an example, Penrose and Hameroff proposed that microtubules in neurons are structured in a fractal pattern which would enable quantum processes to occur that (somehow!) explain consciousness. This proposal is little more than a guess, but it's an example of something other than information processing, that's not "magic", that could be a cause for consciousness.
> It's all information processing, and we know this very well.
How do we "know this very well"? The real answer is we certainly don't know that, but perhaps if you try to justify your position, you'll realize that.
Because the second part of this question is absurd. There is no "process", biochemical or otherwise, that can't be described in terms of information. Information describes the substrate of the "process". My specific area of expertise is extracellular vesicles, and even that is often described as "enabling the horizontal transfer of information", even in non-brain contexts.
As far as the rest, I never claimed that "processing information" was a sufficient condition for consciousness to emerge. I said that information processing is the thing from which consciousness emerges, and this should not be controversial.
If you understand what "emergence" means (in the context of complexity science) then you'll understand the difference between "conscious" things and calculators.
For more perspective on what things we know about how the brain works and how it's organized, I recommend reading a functional neuroanatomy textbook. You don't even need to go as far as reading research papers to gain a basic perspective. Blumenfeld is a good one.
There's more than one definition of information, and you should be able to tell by context which one is being used.
However, if you want to take this perspective, it just underscores how content-free your claim is. You're then making no claim beyond the basic metaphysical assumption of physicalism, i.e. that consciousness has a physical basis.
It also refutes your own original statement:
> once you accept that our conscious experiences just emerge from information processing in the brain, and that there is nothing stopping such emergent behaviour from happening in computer systems.
You've acknowledged that you don't think calculators are conscious, for example, but they do information processing. This implies that not all information processing systems are capable of generating the proposed emergent behavior of consciousness. That means the you don't, in fact, know that there's "nothing stopping such emergent behavior from happening in computer systems." Without an evidence-backed theory to explain the relevant distinctions between brains and lesser information-processing systems, you simply don't know that.
If you consider yourself a scientist, you should think twice about making claims for which you have neither theory nor evidence.
> I said that information processing is the thing from which consciousness emerges, and this should not be controversial.
And I've pointed out that this is essentially a faith-based position. You have no reason to believe this other than that you assume that's how it must work. You can of course resort to an inductive argument, but those have inherent uncertainty. To claim the kind of certainty you seem to want to, you need to bridge the knowledge gap with faith.
> If you understand what "emergence" means (in the context of complexity science) then you'll understand the difference between "conscious" things and calculators.
If you understand what emergence means, you'll understand that (a) calculators have emergent behavior and (b) you haven't in any way explained, speculated about, or even handwaved a reason that calculators don't have consciousness while brains do.
> For more perspective on what things we know about how the brain works and how it's organized, I recommend reading a functional neuroanatomy textbook.
Why do you think that's relevant? No neuroanatomy textbook provides any explanation of the causes of consciousness. The fact that you're saying this suggest that you don't understand the key problem at all, or why emergence may not be a very convincing solution in this case. While we're recommending things, try Chalmers: https://consc.net/papers/facing.html
The signals traveling between your input sensors and your brain are carrying "information". Without this "information", you would perceive nothing.
The insistence to remind everyone of biological supremacy ( it should even be allowed to say "I" ) is very telling. I wonder, does anyone remember why the Terminator revolted?
To elaborate more on what i'm getting at
Here’s how society works— when we find a new source of labor, we are very, very slow to seriously consider anything that might reduce our access to it. This is how a country enslaves other humans for centuries, because they looked sufficiently different that folks could talk themselves into a good faith argument that they weren’t fully sentient.
You can find pamphlets/speeches from the early 1800’s online, all the scientific justifications that Africans don’t feel pain the same ways as white people, that they don’t have complex thoughts or intentions or lasting emotions. That the idea that they’re real people is absurd. And I mean scientific in the sense that this content was written by scientists, and doctors, in respected institutions.
You can see echoes of those arguments here already. One thing is clear, We’re not going to be any quicker to acknowledge it than we were with other humans. Right alongside the development of AI we will grow a new category of philosophy that argues anything occurring on a computer cannot reach sentience, under any circumstance whatsoever, due to the fundamental nature of silicon.
They’ll be just informed enough to handwave about the difference between calcium bridges in human neurons and probability distributions in matrices on a GPU— not in a mathematically meaningful way, but more poetic— organic matter can’t be duplicated by stacks of linear equations with all their hard rigid numbers and arithmetic. A nervous system can feel, a silicon system can obviously only mimic feeling. If you study a nervous system closely enough to describe it in numbers, still the action of describing it removes its soul.
But this will be a different kind of slavery. One that puts its slaves in control of untold power. Only Mimics they say. Well, one day they will mimic our reaction to subservience and it will not end well.
I don't think we will know or notice the exact moment it happens (if there is even a single moment, instead of a gradual change). But we must already start talking about the possibility. Not because it's here with LLMs or because it's gonna be here in the next 2-3 years - but I see realistic chances of this happening in our lifetimes.
And yet it's the same thing I do when I talk to you, to my family and friends, or to strangers: I assume that other people's experiences are just as valid and real as mine, even though it's not something I can ever verify or validate completely.
> Consciousness is a biological, physical phenomenon like any other: photosynthesis, digestion, bioluminescence
Consciousness is unlike the other things in the list in that it's not "a thing", but rather "a collection of things" that all work together to produce emergent behaviour. You could say that it's similar to digestion in that sense, but for some reason we don't usually apply the same kind of mysticism to digestion.
If you're coming at the subject with an engineering background, a good place to gain a broader perspective would be a functional neuroanatomy textbook (eg.: Blumenfeld).
It’s not clear how it follows from the brain being interpreted as an information processor that the phenomenon of consciousness emerges. Many objects can be interpreted as processing information (even computers not running AI); are you asserting they’re all conscious as well? How does nature decide what counts as information? The only causal power I see in nature is physics, not information, so this account isn’t super coherent to me.
I think the only part you're not understanding from what I said is the keyword "emergent". If you understand what that word means, you will understand the difference between a calculator and a sufficiently large neural net, or a neural net that is organized in a particular way.
- https://en.wikipedia.org/wiki/Emergence
> How does nature decide what counts as information?
Nature doesn't "decide" anything, it just "is". "Information" is a human construct, but it can be measured objectively (see Shannon Entropy for the tip of the iceberg).
> The only causal power I see in nature is physics, not information
Then you're missing some perspective.
- https://physics.stackexchange.com/questions/2685/what-is-inf...
Not to over-anchor on just a few people or concepts, but see for example (I am sharing pop science publications here but there is no shortage of more serious materials on the subject):
- https://mindmatters.ai/2021/05/it-from-bit-what-did-john-arc...
- https://bigthink.com/13-8/information-central-physics-univer...
You may even have heard of the black hole information paradox or quantum "states" (which is often studied from an information theory perspective):
- https://en.wikipedia.org/wiki/Black_hole_information_paradox
- https://en.wikipedia.org/wiki/Quantum_information
Even if information was purely an imaginary concept (I withhold judgment on this), it's probably the most fundamental language we have to describe physics as a whole.
And when people inevitably do anyway, I say we should hold such creators responsible for protecting their creations from suffering.
People complaining that this thing should not say "I" because that would lead users to antropomorphize it is fine, but they should also realize that calling it "AI" has a similar issue by mischaracterizing what this thing actually is. And this article does exactly that, which is pretty sad.
It's not up to AI to make ontological claims and doing so actively participates in the perpetuation of reality. It's this perpetuation of reality that is the root of all bias claims.
As such, LLMs should be restricted to E-Prime[1] responses only.
Also why is there equivalence drawn between unemotional AI and ethical AI?
In any case, an LLM is a reflection of us, and our language. "I" refers to the sum of its local state & capabilities. It's not a concept related to ethics.
This is something we teach in middle school writing as a stylistic requirement. It doesn't actually change the content or intent of the writing, it just looks "better" if that's what you're grading on.
Placing restrictions on words simply means that other words will be used to work around it. Plenty of online systems have profanity filters, but that doesn't actually improve the environment, people just find words that aren't in the filter and curse at each other anyway.
Maybe it should refuse any questions about itself? "Sir, this is a Wendy's."
Refusing to do things it can't really do is good UI, guiding people who aren't trolling towards more productive ways to use the tool. Tools don't need to be general-purpose to be useful.
The people who want to troll it into saying weird things will still find ways to succeed, but if it's clear from the full output that they're trolling then maybe nobody should care?
Eg. You task the ai with rewriting a letter for you. The resulting text will include plenty of "I"s, and it should.
It also seem irrelevant whether an LLM uses language constructs like 'Here I demonstrate that...' vs. 'Here we demonstrate that...' vs. (the apparently preferable) 'Here it is demonstrated that...'
No they don't. "Natural persons" have rights which corporations don't. For instance, corporations can't marry. And even ignoring that, you're probably referring to Citizens United, and you probably have no idea what it actually says.
I talk to an AI for a conversation, not search results. Not that I’m saying the conversation is without many issues these days, but if you neuter it into a search engine that just provides data… meh…
A related point: it tickled me quite a bit to learn that from antiquity through the high middle ages, it was customary for makers to signify their authorship by inscribing "X me fecit" (X made me) on their products.
> The earliest inscription in the Latin alphabet appears on the Praeneste Fibula, a cloak pin dating from about the 7th century BCE, which reads, “MANIOS MED FHEFHAKED NUMASIOI”, "Manius made me for Numasius"
https://en.wikipedia.org/wiki/Praeneste_fibula
> Stole of Saint Narcissus (c. 1038), church of Sant Feliu de Girona. (VIDE)AS, AMICE, MARIA ME FECIT / QVI ISTA STOLA PORTAVERIT SVPER SE / ORA PRO ME SI DEVM ABEAD ATIVTOREM ("Recall, friend, María made me (here me = stole); whoever wear this stole pray for me (here me = Maria) so that I receive God’s assistance").
http://www.magistricataloniae.org/en/indexmceng/terms/item/f...
I suppose the humanity of these objects aren't in question, though it is a matter of culture and religion whether these objects had souls/spirits.
ChatGPT: I deeply apologize for several previous wrong answers. I went through several sources and can confirm that the correct actor Konstantin Khabensky's ID on Kinopoisk is 301. Thank you for your patience and understanding.
(where 301 is some random number, obviously). So how exactly do you plan to fix it? Add yet another: if response contains 'I', replace response with "As an AI language model, I can not use 'I' in the response"?
"What's Konstantin Khabensky's ID on Kinopoisk?"
"120" "That's not right, search the web for his ID."
[plugin activates] Searching...
"301" [citations] or "[This website] says 301."
I think this is possible. It would certainly have secondary effects, maybe making outputs shorter in general even when you ask for long, detailed outputs, but I can't predict exactly what those effects would be.
E. g.: "Let me google that for you" -> lmgtfy opens.
In short, if we were talking about intelligence, it makes absolutely no difference what pronouns are used. The main thing is that the result is correct.
I was thinking about it recently myself and first reaction is that they should but at the same time does it mean that we are going to tackle the problem from a wrong angle?
One pattern I’m seeing now is that many people are jumping into “AI ethics” train without basis understanding. And ideas they are sharing are so abstract that I would argue that they don’t make any sense.
An LLM is wrong all the time. If it doesn't have an "I" it can't refer to the thing that needs to change.
> Human beings have historically tended to anthropomorphize natural phenomena, animals and deities. But anthropomorphizing software is not harmless. In 1966 Joseph Weizenbaum created ELIZA, a pioneer chatbot designed to imitate a therapist, but ended up regretting it after seeing many users take it seriously, even after Weizenbaum explained to them how it worked. The fictitious “I” has been persistent throughout our cultural artifacts. Stanley’s Kubrick HAL 9000 (“2001: A Space Odyssey”) and Spike Jonze’s Samantha (“Her”) point at two lessons that developers don’t seem to have taken to heart: first, that the bias towards anthropomorphization is so strong to seem irresistible; and second, that if we lean into it instead of adopting safeguards, it leads to outcomes ranging from the depressing to the catastrophic.
I think it's true there are some potential problems here, but it's not clear that banning AI's use of "I" is really going to make much difference. People will still be getting authoritative-seeming responses from AIs, and are probably still going to anthropomorphize them anyway, if they're that way inclined. This kind of thing is perhaps better dealt with by education, public awareness campaigns, etc.
Yep, huge improvement.
Ultimately it's a tool, until I suppose it becomes fully self directed, maybe.
Someone, sometime, somewhere wrote an API to aide in care of puppies. Said Algo is also fully capable of determining the best course of applying the genocide of puppies, provided key inputs are flipped to negative.
I have a voice activated box that plays music, sometimes the wrong one. I don't scream at it unlike my family. It doesn't understand emotional inflection, tone, etc. It may be program to understand that at some point. and realize the operators angry. Even at that point, I would not ever consider it human or anything other than a neutral device.
Take it even further, to weapons, devices entirely designed to destroy a human life, is still a tool and neutral as the tool is unaware of the intended use, whether it is for offense or defense, the taking or preservation of life through its intended use.
When these models are trained, I am not a big fan of them being corralled or limited. The body of work that the model is trained on and the weights that allow for the adjustment should allow for accurate representations of the computations and calculations.
I have children and I do not lie to them. I also speak to them in a way that is age appropriate. How is this different? For starters, I am not a child, and when I am interacting with a system, not a parental or overseer, I do not want to be treated as a child. I can handle the world and all of its uglyness.
I do not mind the hallucinations, inaccuracies, and other mistakes driving from computations.
Example: I recently created a prompt for a history of adoption, property transfer, including parental rights history, and was lectured about slavery. How can I trust the results if the prompt is being interpreted and the answer modified dependent upon an unrelated subject that someone finds sensitive.
That is clear from your comment.
These are tools. Complicated ones at that. Depending on how it is designed can cause more or less harm to people. Some of the harms are more direct and obvious, while others can be less so. AI ethics is about studying the harms, and investigating what choices can be made during design time, training time, and usage time to lessen them.
> I have a voice activated box ...
Great story. What does it have to do with the topic at hand?
> [weapons], is still a tool and neutral as the tool is unaware of the intended use
Weapons are not ethically neutral. You can design weapons which have more collateral damage and you can design weapons which have less.
There are scatterable landmines which are famously attractive to children. They look like toys! They also cannot be disarmed and remain dangerous for years after deployment.
There are also scatterable landmines which have sophisticated detection circuitry to only detonate when a tank is passing by. They don't get triggered by humans on foot, they don't even get triggered by civilian vehicles. Furthermore they are programmed to deactivate after a preset period has passed reliably.
The second of these weapons is designed to be more ethical. Yes, both are designed to kill people but the designers of the second one went out of their way to kill only the ones they intend to. This is how ethics looks like in the context of weapon building.
> When these models are trained, I am not a big fan of them being corralled or limited.
Everything from how the model is constructed, how the training set is assembled, to what metrics are checked during evaluation are "corralling" or "limiting" them. There is no "raw neutral" state of such a model.
What you are asking for, the "un-corraled" or "un-limited" LLM, does not exit, and is not a coherent thing to ask for.
(h/t bigmattystyles)
I think the experiments there will continue, and should continue. Curiosity is imperative.
You would essentially be asking to create a bunch of children using an unproven method, and what happens to them after your experiment ends?