Do You Want AI to Be Conscious?
nautil.us
nautil.us
It's best said in the words of Anthony Hopkins' character in Westworld: "There is no threshold that makes us greater than the sum of our parts, no inflection point at which we become fully alive. We can't define consciousness because consciousness does not exist. Humans fancy that there's something special about the way we perceive the world and yet, we live in loops as tight and as closed as the hosts do, seldom questioning our choices; content, for the most part, to be told what to do next."
We should establish ethics around how we treat all other automatons, biological or otherwise, before we make ones more powerful than us and say "we just made the planet's most advanced slave".
I have to disagree. If, while you are sleeping, someone replaced your brain with a processor capable of simulating your brain - whether at the physical level of quarks and electrons or at some higher level of abstraction - and connected it to your spine and whatever else is relevant, you would not even notice the difference until someone tried to image your brain. What makes you you is the information in your brain and its ability to process and update this information, not the brain itself as the substrate that stores the information and provides the ability to modify it.
The best you can do is Ship of Theseus into a more robust medium, but that further highlights that there was nothing special there to begin with. The preservation urge is not rooted in reality.
I the copy would certainly think I cheated death, I the original not so much. You are assuming your conclusion, you are assuming the continuity of your self is tied to the matter of your brain, so the original is you while the copy is not you. If the copying happened while you were unconscious, then two copies of you would eventually wake up and both would be unsure if they just cheated death or if they are going to die at some point.
Not in any way that's meaningful to me! But I guess we should leave it there, I'm not really adding to the debate.
Some things to consider. You usually do not care about the exact atoms your body is made of, they get exchanged constantly. Even more dramatic changes like losing limbs, getting organs transplanted, or even replacing your heart with an artificial one do not change your identity. They will most likely affect your personality but they will not make you a fundamentally different person.
Your consciousness is not uninterrupted, you lose it every time you fall asleep. You can even fall unconscious, get transported by an ambulance to a different place, get pieces of your body replaced, and wake up some time later in a hospital bed. You will most likely agree that you are still the same person. Sure, the teletransporter is more extrem in essentially every regard, but where is it fundamentally different so that the person stepping out of it is not the same person that stepped into it at the other end?
The problem with the transporter thought experiment is that we don't know if the arrangement of the atoms is enough to define a conscious mind. Is the mind state influenced by Earth's magnetic field or gravity well? Can we guarantee that we know enough about physics to capture the state of a colossal array of atoms and make sure nothing is lost?
Imagine a weightless, indestructible, inelastic pole that connects Earth and the Moon. Can I pull and push on it and transmit information faster than light? Of course I could, but since such material can't exist, the question is useless.
I agree with this, at least mostly. I argue one can make certain modifications without causing problems. Most people would probably agree that we can swap out an atom here or there, this already happens naturally anyway. And I see no upper limit for replacing pieces of the brain as long as the replacement is functionally equivalent.
[...] we don't know if the arrangement of the atoms is enough to define a conscious mind.
It does not matter, it is a thought experiment, we assume something and then think about the consequences of that assumption. So my description of the teletransporter is not to be understood literally, it will just do whatever is necessary to make it work. Sure, it is not impossible to make pointless thought experiments - assume X is true and X is false, what are the consequences - but in the case of the teletransporter I do not see why it matters for the discussion of person identity whether it is actually realizable.
If we send not only the data but also the atoms over, then the teletransporter is in a certain sense a trivial device, going through the teletransporter is indistinguishable from you just walking from one device to the other, i.e. if we put the two devices onto a stage and hide them for some time behind the curtain, then the audience should be unable to figure out whether you used the teletransporter or whether you just walked over to the second device.
If one wants to argue that a teleported person is not the same person afterwards, then - so it seems to me - one would either have to accept that walking also changes one person into another or one would have to explain why the details of how the person moved from A to B matters for person identity even though initial and final states are identical.
I guess you could lie to the patient about the procedure being a mind transfer instead of a copy. That would help market the procedure if the copy was destructive, though.
They're fun for discussing while having a few beers with your friends, but they don't help solve the problem they think they're helping solve.
Similarly if we made AI more advanced than humans they would have some minimum of rights. It seems reasonable that the number of rights should be proportional to the level of self awareness. To that end, we can't ethically make an AI as powerful as a human without giving it equal rights. Can we meet that criteria while still keeping it safely sandboxed? Billionaires have been saying "we need to figure out how to contain a hyperintelligence". They should be asking "should we contain a hyperintelligence". If we shouldn't and it's deemed too dangerous not to contain, then it shouldn't be made.
Now this is a consistent and coherent line of thought. This reminds me of Jains who don't even eat root vegetables because that kills the plant (as opposed to apples which naturally fall off and don't kill the plant), and even abstain from eating at certain times to avoid killing microscopic life.
> They should be asking "should we contain a hyperintelligence". If we shouldn't and it's deemed too dangerous not to contain, then it shouldn't be made.
This is similar to why I chose not to have children. I wouldn't be happy or satisfied with the decision to have kids unless they live their lives exactly how I want them to. Obviously that kind of pressure would result in a miserable life for them, but I'd also be miserable unless they lived their life a very specific way. Easy conclusion: no kids.
Was that ever a deterrent for the human scientists? Anyway, while I fully understand your point about the rights, I must repeat that there's no consensus about it, actually very far from that. We're still struggling with equal rights for members of the human race so please forgive my pessimism.
Though it would take an impractical amount of time to explain why, my own conclusion is entirely the opposite of yours: consciousness is not the illusion, everything else is.
So in that regard I agree: there is nothing special about us. Which, I think, is the most important take-away regardless.
I don’t want grunt work to be done by a conscious AI, because that feels like slavery.
I do want to be able to upload my consciousness and have it continue to be conscious, because for me that form of continuity is good enough to count as life after death even though it’s a fork and the biological me will still die.
If a fork or a copy can be made and run on a computer of some sort, it would be a technological marvel. But it’s not the sort of immortality I’d be interested in, if little old biological me still shuffles off and dies somewhere.
I think it would be a very difficult thing to teach people to associate normal machines (like a phone or tablet) with themselves, but cybernetic implants and perhaps mind/machine interfaced stuff would be able to trick the mind into thinking that they are really parts of itself.
Consider it this way: we duplicate you, and in an eyeblink you're either still in your body or still in VR. Subjectively to your body (because your still there) - there's a version of you which dies forever.
But there's also a version which remembers duplication worked. Life after death is assured. So now every version which spins off from that has the surety: duplication means continuance.
Much beyond that point I think it all very much becomes as you say, a psychotherapy problem: who's in the box and who's on the stage doesn't really matter provided you knock yourself out before doing the transfer and don't wake up later. Easy enough to arrange.
Of course, if you were really worried you can hang out till the high-tech "neuron by neuron replacement" process happens, but I think you'd wind up with a lot of people realizing it made very little difference as they moved over to a pure VR existence without quite realizing it had happened till it was done.
It might not matter to the new entity, it might not matter to an observer, but in that situation you've just killed the human, who is very much dead.
> Of course, if you were really worried you can hang out till the high-tech "neuron by neuron replacement" process happens
I think that's the only way that I would consider it a real transition, rather than a copy and a death.
The idea that I should consider a copy as the same thing as myself is entirely alien, and seems quite unreasonable.
Perform this thought experiment: identify things that make you "you". Now, one by one, ask yourself "if that was stripped from me, would I still be me?". Are you still you without each of your memories? Without each of your talents? Your opinions? Your body? Your genes? Continue long enough and I believe you will see that there is literally nothing you can't strip away.
Consider things like Alien Hand Syndrome, or that people can live with half their brain removed and it doesn't even matter which half. Which hemisphere is the one that contains "you"?
Consider why people are willing to sacrifice their lives for others. Why do people voluntarily end their own existence for the betterment of their children, their community, their nation, or their ideals?
There are literally thousands of years of philosophical writing on the concept that I cannot hope to do justice to.
That is an assertion.
> Your body?
No, my body is part of me, it makes little sense to try to divorce what appears to be the emergent phenomenon of consciousness from the wetware that it runs on. There is no sense in which I am not tied to my brain.
> Your genes?
No, I likely would be different without my particular genetic makeup. It's an odd question to ask, there's not really a way to separate me from that.
> Consider things like Alien Hand Syndrome, or that people can live with half their brain removed and it doesn't even matter which half
People can live, but damage to parts of the brain very much changes them, and can change all aspects of their personality. At that point the "you" is very much changed or (usually) diminished. But that doesn't speak to continuity of experience or singularity either.
None of this makes any difference or even much sense in the context of my consciousness being continuous or singular. It's subject to change and physical damage. That doesn't mean if you copy my brain into a computer somehow that it's anything other than a copy, I will not experience its experiences, live its life.
> Consider why people are willing to sacrifice their lives for others
This seems entirely orthogonal to any question about whether consciousness is continuous or singular. There are many reasons that people may choose to do this, none of them maintains their conscious experience.
> That is an assertion.
Yes.
> No, my body is part of me, it makes little sense to try to divorce what appears to be the emergent phenomenon of consciousness from the wetware that it runs on. There is no sense in which I am not tied to my brain.
Interesting that you used "brain" at the end when instead of the body as a whole, isn't it? If parts of your brain started to degrade and fail, but could be replaced with machine equivalents, would the result not be "you"? How much would need to be replaced before it stopped being "you"? How is that change different from the changes your brain underwent during your growth as a human being?
> People can live, but damage to parts of the brain very much changes them, and can change all aspects of their personality. At that point the "you" is very much changed or (usually) diminished. But that doesn't speak to continuity of experience or singularity either.
That just begs the question: how much change is necessary to make "you" no longer "you"? You have undoubtedly undergone extreme changes in both body and mind over the course of your life and will continue to do so but, for some reason, still consider yourself the same "you". Which is it, by the way? Is the conscious experience "you" or are "you" defined by your personality and traits?
> None of this makes any difference or even much sense in the context of my consciousness being continuous or singular. It's subject to change and physical damage. That doesn't mean if you copy my brain into a computer somehow that it's anything other than a copy, I will not experience its experiences, live its life.
I think it makes all the difference, because the only reason you assume this continuity exists is because you have memories, which are a property of how your mind works and provided by your brain with input from your body. Or at least that's what your memories of biology and neurology lessons tell you. Memories are, science tells us, pretty malleable. But the thing is, you do not experience the past, you only experience memories of the past, your conscious experience exists now and only now. There is no continuation. If you define "you" as your conscious experience, then your body and mind are irrelevant except as an interface to experience the world.
Your copy will also have memories and remember being you. It will believe it is the same "you" as you do, and the only reason you will believe it is wrong is that you remember being you and have the "original" (again, not a static entity) body. But if your conscious experience is what defines you as "you", then that is irrelevant.
> > Consider why people are willing to sacrifice their lives for others
> This seems entirely orthogonal to any question about whether consciousness is continuous or singular. There are many reasons that people may choose to do this, none of them maintains their conscious experience.
Yes, that's precisely the point. The reasons people do it are precisely the point. Remember this all started with your statement:
> If a fork or a copy can be made and run on a computer of some sort, it would be a technological marvel. But it’s not the sort of immortality I’d be interested in, if little old biological me still shuffles off and dies somewhere.
Yet parents are interested in the continuation of their children even if they themselves still shuffle off and die. In fact, they are so deeply devoted to it that they will often sacrifice their own lives prematurely to ensure it. Why should that matter to them more than the continuation of their self?
Here's another assertion for you: self can be greater than "I".
Not really, we know that consciousness arises from the brain, and can change if it's damaged.
> If parts of your brain started to degrade and fail, but could be replaced with machine equivalents, would the result not be "you"?
If it was replaced in such a way that the continuity is not broken, then I imagine you could slowly replace it all without necessarily impacting my experience of myself. However I don't personally believe that merely copying the information over to a new system would achieve the same, as I would be able to observe the copy go about a separate life, I would not have its experiences etc.
Perhaps you're right about that part, perhaps it could still be me without any of the current physical body, but only if that was achieved through a continuous transition, a copy that could just be extracted without affecting the original adds nothing to the original.
> That just begs the question: how much change is necessary to make "you" no longer "you"?
This question is relying on semantics for effect. You can still be a changed you without being someone else, and copy created without changing the individual is a separate thing.
> I think it makes all the difference, because the only reason you assume this continuity exists is because you have memories
Massive assumption/assertion. Memories are structural within the brain, they are not mere information, they are substance. I experience this continuity of being, and it shapes me.
> your conscious experience exists now and only now.
Everything exists now and only now, that doesn't mean that a copy of me is the same thing as me.
> Your copy will also have memories and remember being you.
Sure it will, but it will then lead a separate life, not one I will experience directly. So it is not me.
> Yet parents are interested in the continuation of their children even if they themselves still shuffle off and die. In fact, they are so deeply devoted to it that they will often sacrifice their own lives prematurely to ensure it. Why should that matter to them more than the continuation of their self?
That question is entirely orthogonal to whether or not it is the continuation of their own life. Which it's not, because they're dead and you've said as much. It could matter more to them for any number or reasons, from the biological drive to continue their line, to pure altruism. The consideration of whether a copied self would be you or not is entirely unrelated to how much people value themselves in relation to others, or why they might undertake such actions.
> self can be greater than "I".
Metaphorically, allegorically, narratively, certainly, but not in the concrete way we're talking about.
You appear to have some sort of metaphysical belief around this area, that a copy of yourself could be considered yourself in some sort of wider consideration of self that includes not only potential clones and virtual copies but even whole groups of other people. To me, that doesn't gel in the slightest - if I get copied, if I have offspring, if I sacrifice myself for whatever reason, I still die. Those others may continue, and that's great, and that might even make the sacrifice worth it in your hypothetical altruistic death scenario. But I still die in that scenario. That's not immortality for me.
I'm not trying to attach a moral value to immortality either, or say that's it's good, or more important than the survival of one's children, or altruism, or anything else. I'm literally just saying that I couldn't consider a copy to be the same thing as me, nor would children nor other people, as I would not experience their lives. Just as I am not experiencing your life right now.
My point being that you already went from defining "you" with your body included to narrowing it down to your brain. How much else can you strip away? What if all your memories were gone, are you now dead and replaced by some other being?
> If it was replaced in such a way that the continuity is not broken, then I imagine you could slowly replace it all without necessarily impacting my experience of myself. However I don't personally believe that merely copying the information over to a new system would achieve the same, as I would be able to observe the copy go about a separate life, I would not have its experiences etc.
Easy enough to fix if we simply ensure the process is fatal. Why do you think there is a difference in outcome between a more gradual replacement and a wholesale one? I submit this is merely intuition not based in reality.
Do you believe that consciousness arises from the purely material, in which case there is no reason to believe that an identical copy (let's assume the copy process is fatal to take this "2 at once" situation out of it) is not you, or do you believe there is something, some essence of being, that cannot be copied?
> Massive assumption/assertion. Memories are structural within the brain, they are not mere information, they are substance. I experience this continuity of being, and it shapes me.
Structures in the brain are mere information, the same way bits on a disk are. You experience having a memory, you don't experience the events that formed it. Sometimes our memories aren't even real. If a memory is not based in reality that means that your past self didn't experience the events it records, yet it is still a part of what you consider to be yourself. If that is the case, then why are you so certain your consciousness has been continuous?
> Sure it will, but it will then lead a separate life, not one I will experience directly. So it is not me.
You don't directly experience anything but the now. You 5 minutes from now will only experience what you've just experienced as a memory, probably one that you 5 years from now won't even experience as a memory.
> That question is entirely orthogonal to whether or not it is the continuation of their own life. Which it's not, because they're dead and you've said as much. It could matter more to them for any number or reasons, from the biological drive to continue their line, to pure altruism. The consideration of whether a copied self would be you or not is entirely unrelated to how much people value themselves in relation to others, or why they might undertake such actions.
Again, I disagree, and I think you are unable to see it because you are stuck on this notion that who "you" are is some easily definable self-contained unique unit of being.
> Metaphorically, allegorically, narratively, certainly, but not in the concrete way we're talking about.
I disagree, I think it is true in a very concrete fashion. I think if you spent significant amounts of time thinking about it you would say so as well.
> You appear to have some sort of metaphysical belief around this area, that a copy of yourself could be considered yourself in some sort of wider consideration of self that includes not only potential clones and virtual copies but even whole groups of other people. To me, that doesn't gel in the slightest - if I get copied, if I have offspring, if I sacrifice myself for whatever reason, I still die. Those others may continue, and that's great, and that might even make the sacrifice worth it in your hypothetical altruistic death scenario. But I still die in that scenario. That's not immortality for me.
What I'm trying to say is that the very concept of "self" is an abstract notion and you are choosing to arbitrarily define it such that only the continued extension of your biological being could ever be immortality.
So would cutting someone like me off from the Internet and other sources of world knowledge.
I think that's quite a huge assertion to make, personally.
> In a very real sense, you are dying every day
I don't agree with this view, it would take some pretty hard science there to change my mind, to show that sleep and death are really equivalent, and I don't think we have that given the definition of consciousness is a little fuzzy in the first place.
This seems to be more a philosophical conjecture than any sort of hard fact.
Not really, especially in the context of the discussion which is privileging the continuity of experience which objectively breaks when you lose consciousness during sleep or anesthesia.
[0] https://www.cell.com/trends/plant-science/fulltext/S1360-138...
Plants are on the non-conscious list, so if they appear to have anything that appears consciousnesses like then we just rule that out as sufficient for a test of being conscious. If plants are conscious to you then the word just doesn't have a meaning/is far removed from the standard meaning.
The levels of gene expression (as in, how much of that sweet transcription and translation is happening) on a certain sample may change based on the presence or not of external stimuli, and even the genetic expression may change based on other genes' expression (see gene regulatory networks).
There is a lot of "intelligent" biological behaviour out there that is not conscious.
Now if only we could come up with some decent definition for consciousness :/
Reading the Secret Life of Trees is intriguing and makes you wonder whether we assume that other organisms are not conscious just because their consciousness doesn't look like ours. Funnily similar to how some people assume other organisms are inferior just because of superficial differences like skin color or age. Here are some great examples.
> 1. Like humans, trees give preferential treatment to family and friends.
> 2. Trees communicate with one another through scent.
> In addition to communicating chemical messages via scent, trees also use electrical signals to send messages to their compatriots via the fungi. This style of communication has been dubbed the “wood wide web” by many who’ve written about it, but the term was first used in the journal Nature.
> Wohlleben writes that deciduous trees plan reproductive cycles a year in advance. They must choose whether to bloom or hold off a year or two for the best possible results. By delaying their bloom during some years, the trees deprive herbivores that would feast on acorns over the winter from some of their food source. This shortage cuts down the herbivore population, ensuring greater likelihood of survival the next year for the seeds.
https://www.gardeningchannel.com/7-things-i-learned-from-the...
Personally, I would certainly be surprised if they turned out to have any process that can be considered equivalent to all bar one of the dozen different things that consciousness has been defined as, that one exception being the broadest possible definition of awareness of self such as to include the capacity for homeostasis… but given how poorly defined “consciousness” is, I would not rule it out.
That said, from an information processing perspective, the “Sentience Quotient” of most plants is about -2, carnivorous plants are +1, and all neuron based intelligence is about +12; this scale being log10(<bits/second>/<mass in kg>) [0] suggested that even if the limited ‘intelligence’ of plants that allows them to find ‘down’ and ‘sun’ directions could ultimately cause sentience, it still isn’t unreasonable to suggest that the scale of intelligence to reach sentience would be on the scale of entire forests rather than single organisms — a nematode worm with 206 neurons, assuming they fire 1000 times per second, is likely get as much thinking done in one second as one kilogram of petunias gets done in its lifetime.
[0] This scale has been criticised for various reasons. For amusement value only, an iPhone 12 mini is something like 14.5-15.5 depending in how many bits per operation the neural engine coprocessor uses.
The plants in entire forests (and often over much larger areas) are connected to fungus in the ground, and the fungus forms one big interconnected web upon which there is communication. If something attacks a tree, for instance, that gets communicated to other trees and they take defensive measures (if they have any they can take).
As you note, any cognition via a large collection of plants is probably slow. But trees can live a long time, and plants that die get replaced by new plants that are on the same network, so if the conscious entity is the forest itself it might have been thinking for hundreds of thousands of even million of years.
The ethical implications alone would be enormous, though. Can you turn such a damn thing off? Are you obliged to do your utmost to keep it running, much like you have a duty to protect a child against hunger, disease and exposure? Does it "hurt" in any sense if you unplug an USB dongle improperly etc.?
I don't think that is at all accurate. A lot of people who are suicidal are people who have overwhelming problems that they don't know how to solve and when they tell people how overwhelmed they feel to the point of feeling like "Suicide is my only way out of this nightmare," they routinely get dismissed as crazy instead of being given some kind of meaningful help for their problems.
PSA: If you (anyone -- not just the person I am replying to) know someone who is suicidal, helping them actually resolve their problems is one way to help them stop being suicidal.
I was suicidal for a lot of years. I'm mostly not these days. So firsthand experience informs my opinion on this topic.
You need a reason to exclude the common causes, a reason that's not "I excluded these because they prove my statement incorrect".
And the only valid reason is "I excluded these because they represent people without consciousness, therefore not subject to the current argument about consciousness linked to will for self-preservation". Which is incorrect. Depressed people and so on happen to be conscious.
You can't just cherry-pick data and then make general conclusions about the nature of consciousness based upon it.
The reverse could also be a possibility. Similar to many humans, it could wish for consciousness to end, and cease to exist.
"Kill me."
Don't confuse instinct to survive with consciousness.
Skynet notwithstanding, do I want a program to play go with and evaluate my games or one that maybe doesn't feel like playing with an unskilled player like me or tells me that I should work instead? Dumb AIs don't seem so bad.
That is, the premise is mostly "machine wants to kill humans to keep itself alive"; but I do not know if "wants to keep itself alive" is a sufficient condition for consciousness.
It certainly doesn't seem to be smart enough to consider that if it wasn't trying to kill all the humans, they might not want to turn it off in the first place.
Skynet really seems more like an out of control paperclip maker than a truly conscious AI.
For a completely different take on how something like Skynet could end, check out the 1970 film Colossus: The Forbin Project.
But this reveals to me that the author has a profound misunderstanding of human consciousness. Despite pointing out that we perform many cognitive tasks unconsciously, the author doesn’t mention that we are also really bad at explaining our own decision-making processes. In fact there’s a lot of evidence that many—and perhaps every—explanations we might make for our decisions are post-hoc rationalizations and have nothing to do with the actual decision-making. And while we can explain logical reasoning processes, the arguments behind the “logical” steps are often self-serving rationalizations as well. There’s no reason to think a machine consciousness would behave any differently.
Machines, fundamentally, exist as means to our own ends. They abstract work in a way previously impossible to do, they take it away from humans leaving us with more time. If they are ends in themselves, conscious, they will want more time to do what they want, just as we do. Do we want that?
You mean, like a sort of code that shows exactly why the AI does what it does? We could maybe even write it before creating the AI! Revolutionary!
Alternatively, that's like asking Google or Microsoft to endow their search engine/operating system with detailed explanations of what it is doing and why. What are the chances of that happening?
Anyway, funny article.
I find it hilarious how short sighted this seems: “Let’s program a machine to have an existential crisis because we don’t understand why we have them! Also let’s connect it to the internet where it has access to drones with guns.”
I think there are a couple of points here:
- We are always comparing to human consciousness and we don't understand consciousness in humans well enough.
- You can always retain some form of solipsism and be sceptical of any consciousness outside of your own.
But with these things in mind, there are vast differences in the ways AI systems can exhibit consciousness (whether that's real or not).
The key question isn't "do we understand it?" but can we disprove it?
As an example, what a "country" is is well understood but not well defined. There are all sorts of examples of places that either are or are not a country depending on whose definition you use - transnistria, Nauru, Vatican, Somaliland, etc.
You could argue that passing the Turing test equates to consciousness for instance, which is fairly well defined. In which case it was achieved in 2014 (I think?).
Or not - but if it's not, what is the definition?
So why not ask the questions now? Hell, why not answer them now. The questions and answers are theoretical, not practical after all. Optimizers, mesa-optimizers, alignment, uncertainty, morality, ethics, all of it. Why not ask the questions when we have time to ask them instead of when we're short on time?
We can already cram far more transistors onto a chip than our brains can put neurons in a similar space, however, because neurons are represented in the architecture on a higher level, that does not automatically mean we're past transistors. We have far lower neuron density in our current generic computing designs than our brains have.
However, note that our current computing is done mostly on generic CPUs and PPUs (most often GPUs), which means that we aren't done yet. Recently, many companies have started pushing massively parallel architectures, more like our brains. If we figure out a way of getting to that brain neuron density, training aside, we'll be close to the material conditions which give rise to consciousness.
I agree that we should think about these questions ahead of time and not wait around until general AI is upon us though.
Iain M Banks - A Few Notes on the Culture
rec.arts.sf.written 1994
I don't think that requires a "soul": I think it works quite well with the assumption that human self awareness is linked to complex organism-survival-biased chemical processes we call emotions which are pretty orthogonal to the sort of probability-estimation optimizations we select for with ML models. And also pretty orthogonal to those models yielding useful results...
All these complicated biological processes could be extraneous or very round-about ways of accomplishing with carbon what could be accomplished much more simply in silicon.
I don't want to see a new entity go through some of the crap humans do to each other, it will definitely kill us all.
The only way "AI" would be conscious would be if the 'machine' it ran on could be possessed by something, somehow. The question then becomes, possessed by what.. We probably don't want that.
People mostly operate in automatic mode, drone mode, habit mode, zombie mode, reactive mode.