Hacker News Reader

Comment by dodo53 | Hacker News Reader

One thing I always think about brain emulation/mind uploading/strong AI research is we need an legal/ethical framework in place before we get there.

I still think it's far off, but not everyone does. If you believe consciousness is a property of running a brain or sufficiently advanced brain model, at what point are you allowed to stop running your model and it's not murder of a sentient? At one point can a model claim rights?

Especially as their goal is to use it for drug tests.

Also, can you simulate a brain without a body or is say hormones/blood sugar/nervous system not easily separable?

View on HN

Of more concern to me would be control of my own consciousness uploads.

I don't believe that hell exists now, but it would certainly be possible to create a technological hell if brain uploading and simulation is possible.

Torture is already horrific, but at least you can only be tortured to death once. With brain simulation, it would be possible to torture someone again and again and again.

If copying everything wholesale is possible before brain science is able to fully understand every detail brain scientists want to write a thesis about, then using systematically modified copies for "knock-out" studies will seem like a very tempting methodology. Also gives researchers numerous virtual insane patients to try virtual drugs on.

Or perhaps something with some clear virtuous intent (and nothing particularly tortuous)... what about running Monte Carlo simulations using many identical instances of the same copy to find which pedagogical method(s) and steps best educates someone about a difficult to grasp subject? Even when the creation of the virtual minds is only of good intent, one can wonder about the ethics of it. What if anything is owed the legion of virtual students after the classes are done? I suppose some of them could be given some other useful employment to earn their watts.

Or even without maliciousness - someone could screw up a simulation such that the simulated mind is in pain for a subjective eternity

There's a light at the end of the tunnel. It's likely whatever energy source for the simulation would be disrupted by some catastrophic event. Even without sentient intervention, over tens of millions of years you've got starvation of some resource -- even if it's something like fusion or the raw resources needed to build and maintain "renewable" energy. Over hundreds of millions or billions of years you've got asteroid impacts and cosmic events, eventually ending with the heat death of the universe. Not so bad after all, just tough it out soldier.

That's not really much of a consolation.

I don't believe/think that consciousness/awareness is a property that can emerge from running a brain or a brain model.

But, even if it were: a computer simulating my brain will not be me; I will not experience what it experiences.

Suppose someone came from the future and claimed to be "me"; that is, 20 years from now I will go to the past; or something like that. Let's call the future me F and the present me P. I'm P, when P sees the color red, it's me who's seeing it. But, when F sees the color red, P doesn't experience the same thing. (it will experience it 20 years later, when it becomes F).

So if a clone of me experiences something; I won't experience it.

So if you upload my memories to a brain and then torture that brain, you wouldn't be torturing me; you'd be torturing a machine that happens to have my memories. In the same sense that if you clone me then torture my clone, he will experience the pain; but I won't.

If I do experience the pain of my clones, then we (me and my clones) must be telepathically communicating, or something like that, which, while philosophically not impossible, it's not compatible with physicalism, which you have to accept if you think consciousness can arise from running a brain model. I say telepathy is not compatible with physicalism because telepathy implies a super-natural communication between minds (where minds are implies to be super-natural entities).

The idea that people who aren't so sure about this as you are have isn't that telepathy is going on. It's that if this sort of mind-state cloning is going on, you might no longer have grounds to expect to wake up as the physically original instance of yourself with 100 % certainty after the cloning procedure.

People are made of atoms, and so far we haven't found any magical quality beyond physically stored memories that gives us reason to think a person at one point in time is the same as a person in another point in time, including ourselves as the person. With the mind-state cloning going on, at some point in time there is going to be one present person with a past person's memories being unsurprised at not having ended up as an upload, and another present person with the same past person's memories being quite surprised at ending up as an upload despite remembering having been a physical person who expects to continue living as a physical person. I'm not quite sure what kind of definition of self would let the past person maintain a justified 100 % expectation that they will end up as the non-uploaded version of themselves in the future.

It's about empathy. I can't be tortured that way, and neither can you. But a copy or a construct could be - and many of us find the thought just as distasteful as if it were a flesh-and-blood person.

If the simulation could talk to me, I'd rather it be treated decently.

This being said, I think there are ways to go around this problem. There are many ways in which my brain could be disabled so I wouldn't mind it being "tortured" or experimented upon. Consciousness is an emergent property - take away a few of its building blocks and it disappears.

Do you think it is OK for F or a brain mdoel to experience pain? As long as it is not you who is feeling the pain, it is moral acceptable? Is that what you are saying?

Well no, of course not! (assuming a brain model does experience pain)

I'm just saying if this theoretical "virtual hell" does really happen, it will be a computer hell, not a human hell.

Although I don't actually think a computer simulating a brain model will have consciousness. However complex that simulation is, it's nothing more than electrons flowing through a circuit, and this flow has no intrinsic meaning; I have no reason to believe consciousness can arise from such a thing.

He's only saying that a copy of a brain would create a separate entity, and that separate entities won't share experiences. Like how identical twins have different experiences. There were no claims about morality of using brain models for testing.

"Of more concern to me would be control of my own consciousness uploads."

I don't bring it up very often, but this is actually one potential positive case for DRM. I want control over my brain state. If Hell does not exist, Man may very well create it for himself. I've come up with some nasty tricks you can play with brain states that I don't even want to record to electrons, though the odds of nobody else ever thinking of them are zero.

> I want control over my brain state.

But who is "I"?

If there is a meat you, and a digital copy of you, does MeatJerf get to control DigiJerf's distribution?

How about one MeatJerf and two copies, DigiJerf1, and DigiJerf2?

How about when MeatJerf dies, leaving behind two digital-orphan-clones?

While that is often an interesting question, it isn't very interesting in this case, because for all J such that J is a Jerf, J demands control over J's brain state.

I think it becomes interesting when one of the Jerfs demands something different from the rest.

Also not really that interesting; I am not saying that J's only demands are control over his brain state, only that J has that demand.

Actually I've spent a lot of time contemplating this sort of thing. One of the things you must be very careful not to do is to give in to the temptation to continuously shift frames every time you arrive at anything resembling an answer to a question. The invitation to "ah, but what if...?" everything may feel all profound & stuff, but it is a path to persistent incomprehension, not understanding, because you can "what if...?" your way to anything, and therefore it has no actual information content.

Sorry, I'm still not following. What exactly do you mean by 'control over his brain state' ?

J is just an abstract concept. The concrete demands come from individual Jerfs which could contradict each other.

I must be missing something.

Perhaps a concrete example would be helpful?

I think you might be missing the point of whether it be ethical, or right, or humane to inflict any form of torturous experience on the conscious copy of a person.

Regardless of the hierarchy of the consciousnesses - if they can still feel pain subjectively, where does the line get drawn as to what is right?

If they can independently think/be sentient then they have value to the universe (potentially) and it would be wrong to murder them.

"I've come up with some nasty tricks you can play with brain states that I don't even want to record to electrons, though the odds of nobody else ever thinking of them are zero."

Even before we get to minds on chips, there are nasty mind tricks coming ever closer to our reach. Imagine how much more successful an "investigator" at Guantanamo Bay could be during "interviews" if they (unbeknownst to the person being interviewed) had a "pleasure center" or "warm friendly feelings" remote control in their pocket. That just requires advanced brain prosthetics and some surreptitious surgery. An advanced form of putting a euphoric drug in someone's iv when they see you, so they come to like you a lot over time.

Doesn't even need to be an iv. A spray will do: http://en.wikipedia.org/wiki/Oxytocin#Drug_forms

(in fact, I'm guessing that oxytocin is what this guy is spraying around so that the lions he's playing with don't decide to eat him: http://www.youtube.com/watch?v=-kjWBgA81LM see minute 0:57)

That technique doesn't even require drugs, and has been used for millennia. In fact, drugs can tip potential "interviewees" off that something is not right.

The latest Culture book from Iain M Banks has a war in virtual heavens over the existence of virtual hells:

http://en.wikipedia.org/wiki/Surface_Detail

Yup, I should have mentioned that.

I thought I arrived at this idea independently, and then was surprised to see that it featured in the book.

OTOH, considering I've read every Iain M. Banks book, it's a bit silly to talk about arriving at a conclusion independently (rather than having followed the same path).

In "A Colder War" by Charlie Stross, Cthulhu runs the souls it eats:

http://www.infinityplus.co.uk/stories/colderwar.htm

Edit: this is a rather splendid wee story where "weakly godlike agencies" are treated like any other strategic cold-war weapon, leading to an even scarier version of the Cold War.

You could do that to someone already with the aid of drugs. Stanislaw Lem's The Futurological Congress is an absurdist novel which explores these ideas.

It reminds me of the short story http://en.m.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_S...

Reminded me of a short novella called Deus X by Norman Spinrad:

Norman Spinrad explores the depths of what it means to be human; more accurately, he delves into the nature of the soul in our increasingly computerized technological age. Featuring a poignant new Afterword by Spinrad, this reprinting of one of Spinrad's most cherished works is more timely than ever before. Can human consciousness exist within the framework of an electronic "brain" and still maintain its humanity? In DEUS X, a dying priest's consciousness is uploaded into the most advanced computer of the day - and what ensues is a thought-provoking, entertaining and overly intriguing clash between the various characters surrounding the experiment, a female Pope and a computer guru who'd rather be sailing and smoking pot, for example.

http://www.amazon.com/Deus-X-Norman-Spinrad/dp/0979477018/re...

On the bright side, it's also possible to create heaven.

At least people wouldn't voluntarily sign up to live in hell forever. Virtual heaven might be a Great Filter -- http://en.wikipedia.org/wiki/Great_Filter .

GHOST IN THE SHELL!

> I don't believe that hell exists now, but it would certainly be possible to create a technological hell if brain uploading and simulation is possible.

Bostom's simulation argument says we're almost certainly living in a computer simulation. This opens up the doors right now for posibilities like "hell." Life after death (with memories, etc) becomes possible...all kinds of things. Scary, really.

Wow that's fucking sick.

This is a great question. I was put under for a surgery at one point. I was effectively "dead" from my perspective during the duration of the surgery in that no time seemed to have passed between when I was put under and when I came t. My consciousness was "off" in a way that is not when merely sleeping. It seems like this is a parallel for unplugging an AI, but then you have the problem of the AI losing whatever is in RAM.

The non-humanness of an AI might slow the realization that creating an AI should probably embody the same responsibilities as having a child.

Technically, when you're in deep (non-REM) sleep your conscious is also turned off.

yes but during deep sleep you still sense the passage of time. when you wake up after general anesthesia, it's like changing channels. very weird.

My most refreshing sleep feels as though I have skipped forward from night to morning without the tedious period of time in-between.

There were times in my life when I woke up (from normal sleep) thinking I didn't sleep at all, while in fact I slept for hours and hours.

I don't, when I sleep I die. (Usually)

IMHO, relevant reading would include The Singularity Is Near by Ray Kurzweil and Altered Carbon by Richard K. Morgan.

I bet everyone here knows the former book. Kurzweil's exploration of the ramifications of thinking machines is fascinating, but his response to the question of ethics can best be summed up as between "meh" and "we'll cross that bridge when we get to it."

The other book takes place in a future where consciousness can be digitized, and mankind has already surpassed the threshold of computing power represented by the ability to simulate/host consciousness. The fictional treatment of this technology and its impact on society -- especially on crime and law enforcement -- is fascinating.

I was about to mention Altered Carbon, glad I'm not the only one who has read it. He did an amazing job of exploring many of the far-reaching impacts of being able to transplant consciousnesses.

To your examples of crime and law enforcement I'd like to add religion and drug use as well.

(minor spoiler alert)

All that's left of Christianity is basically a small fanatic cult, and drug use is far more casual and widespread.

If your body is almost disposable and death isn't necessarily inevitable, then I guess it makes sense things like this follow.

Although not nearly as fast-paced and cinematic, the kinda-badly-written Permutation City by Greg Egan does a much more thorough job of exploring uploading and simulation, imho.

Morgan is a wee bit oversimplifying certain aspects, in particular the tech is almost too nicely packaged and very "physical". Which is certainly a possibility, but Permutation City crosses the speculation with the distributed computing trends we are currently experiencing.

Thomas Metzinger addresses this exact question in his book _The Ego Tunnel_ ([1] a great non-technical overview of his work in Philosophy-o-Mind/Neuroscience), and he comes down strongly on the side that any artificial generation of "consciousness" would be reckless and could likely increasing the amount of suffering in the world.

[1] http://www.amazon.com/Ego-Tunnel-Science-Mind-Myth/dp/046504...

That sounds really interesting, thanks for link.

Happy to share. Metzinger takes a hard line on strong materialism and against the existence of the self. For a more general overview of current strains of scientific thoughts on consciousness I'd recommend _Conversations on Consciousness_ [1] which interviews leading researchers who are representative of the varied approaches to the field.

[1] http://www.amazon.com/Conversations-Consciousness-Minds-Thin...

This is great science, and I'm sure we will learn a lot about brains, but there is no way a simulated brain will have any kind of consciousness without a simulated body as well. Without a body and an environment to interact with, the brain is like a computer program with no data to run on.

Experiments have shown that if you yoke a paralyzed cat to another cat and let it experience a complete array of visual inputs, the yoked cat will still be completely blind. Interactive embodied experience is the foundation of cognition and consciousness. A brain in a vat is as empty as a computer without i/o.

It would be interesting to have a full-scale simulation to learn about how activation propagates, how different systems interact, yada yada yada. But until you give it a body there won't be any ethical issues.

Processes in a computer can create input/output between each other. If the brain is just another program, why couldn't some other program running inside the same computer send it inputs?

It could! But simulating the human body, not to mention the environment a human being grows up in, is a much harder task than simulating the brain.

But the brain exists in a certain state at any given time, and this state theoretically can be uploaded (from some individual person) to a computer simulating a brain so that we don't always have to "grow" the computer brain.

Your brain wouldn't be worth much without your body. You'd be surprised how much your body does. Think of typing for example. You couldn't do it without the exact nerve/muscle pathways in your hands. Hook up robot hands (even crudely modelled after your existing nervous/muscle/bone system in your current hands) and you'd basically have to relearn to do anything. You'd be a baby again.

I think you'd be surprised how much this applies to. Speech, even things like counting we rely on our bodies for.

That's the opposite of my understanding; Sandberg and Bostrom discusses this and at least as far as compute resources go, simulating the brain is by far the harder problem.

http://www.fhi.ox.ac.uk/selected_outputs/fohi_publications/b...

That may be true of the body, although our nervous system in our body is quite large, not to mention the spinal chord. But it's certainly not true of the developmental environment. For one thing, simulating that would require simulated thousands of other human brains.

`Also, can you simulate a brain without a body or is say hormones/blood sugar/nervous system not easily separable?`

From my basic understanding of the nervous system, to simulate anything approximating normal function, it would be necessary to emulate the body to some degree. There's just too much interdependence with chemical reactions taking place outside the brain for a model to be of much use in isolation.

That is important, but extremely challenging to execute for two reasons.

1. Historically, technology has outpaced the legal framework. Look at copyright law, software patents, and internet commerce for some examples.

2. It is next to impossible to predict the impact that "Strong AI" or a "Singularity" would have on society. Science fiction literature is filled with thousands of different scenarios. Do we expect Congress or another legislative body to create a framework based on each potential manifestation of strong AI, on the 0.01% chance of it occurring in the next 10 years?

Food for thought - there's a chance that the first implementations of Strong AI occur as a result of a public and government-sponsored research program. There's also a chance that they will come about by a small team of dedicated researchers who will use the technology to their (or its) own ends, legalities and ethics be damned.

Yes, I think my current thinking is: this should be considered. But I'm also cynical that lawmakers could get it right, or that the laws would be followed. Strong AI would presumably have military applications too, or could allow rogue nations to outcompete economically if they had 'friendly' strong AI.

As, as long as most of the population / electable people are religious, probably the idea cannot be debated (as many religious would say consciousness cannot be simulated).

"Strong AI would presumably have military applications too, or could allow rogue nations to outcompete economically if they had 'friendly' strong AI."

If you wish to explore this further, you may like to read the books behind the Halo video games, they're surprisingly good. For example, one of the books has an AI character, a cowboy, who controls the agricultural systems on a food-producing planet called Harvest.

Of course, there's also Cortana, one of the main characters from the Halo universe:

http://en.wikipedia.org/wiki/Cortana

> Also, can you simulate a brain without a body or is say hormones/blood sugar/nervous system not easily separable?

Even if those systems are integral to brain function, I'd assume that they are easier to simulate than the brain itself. Is it important to build a "brain simulation" instead of a "full body simulation"?

I agree you could just go on to a full body simulation - I guess you have to somehow decide what level of accuracy is needed for the key features. You're not going to be able to do chemistry-level simulations of your whole body, so you'll have to make higher level simulations of different parts (like here's a function that describes insulin reaction as a response to blood sugar function)

You raise a great point. Definitely agree on the fact that legal / ethical frameworks are required.

However, it might be very hard to do without knowing what the emulated "being" looks like. Its hard to imagine placing legalities and ethics on something whose powers (and possibly, limitations) are absolutely unknown. Consequently, the law might be misplaced, restrictive, or downright abusive depending on what it turns out to be.

Having said that, I agree with your broad point - having that advanced technology without the social aspects to deal with it might be a nightmare!

Indeed, there are a variety of safety questions as well.

If we were simply emulating a human brain, we would not know how intelligent a system we'd create was or whether that system was sane.

Unless you attach arms with machine guns to your supercomputer, I don't think safety is much of a concern, really.

These questions are important, but I doubt it's best to define the laws before developing the technology, if the risk is on the order of killing a person per upload.

Agree on the ethical dilemmas. Imagine being the consciousness in the first few actual working versions.

Technology can bring us literal heaven and hell.

A small room several paces across, a green floor, blue walls and ceiling. A dim bulb overhead just out of reach. Beyond the walls, nothing; void without even a large expanse and stars to ponder. I've had a couple of lucid dreams end in that room, with the walls seeming to softly vibrate, due I suppose to the instabilities of my brain's simulation of the geometry. Deep despair threaten when I contemplated being stuck there, but I always had the hope that I could wake up and get back to this nice stable world of "reality" that we share.

This is not about making a machine that thinks, rather making a useful large scale model of neurons so we can test how they respond on a very large network scale. Robots will not rise out of it. Think of it like an expensive telescope; do we need an ethical framework for that?

Assuming you're a materialist, a fully functional running model of a brain at the neuronal level would necessarily give rise to consciousness, and should be treated with the same ethical rules as biologically generated consciousness.

It will still be a machine, won't be able to kill you unless you give it a gun. Our ethical rules generally show some respect to animals, because we can empathize with cruelty against them, but even that is relative (we 're not so sorry when we eat them). I see no reason why we should preemptively allocate ethical substance to a machine that thinks. Cruel as it may sound, if a robot is mean to me, i will unplug it.

All this talk about strong AI sounds more like science fiction fantasy. We need to realize that this simulation, even 10 years later will probably be extremely slow, taking minutes or hours to simulate 1 msec of brain activity (it would actually be interesting to have an estimate of the computing power that will be required).

> We need to realize that this simulation, even 10 years later will probably be extremely slow, taking minutes or hours to simulate 1 msec of brain activity

This is just a difference of degree.

How stupid would a real-life person have to be before you could put a bullet through their brain, effectively unplugging them?

How do you know that you are real and actually not a stimulation by some kid in another very advanced world? Do you like to be tortured by this kid, due to the lack of legal entities in that world that protect your consciousness?

If it's a simulation, that's also the only representation of the world i can have and can act upon. So i choose to ignore that possibility as inconsequential to existence and take the empirical stance of accepting the world because my senses tell me so. The alternative would be to wait forever for the simulation to end? The great thing about the brain as a machine is that it allows you to doubt about everything but it doesn't mean that all doubts are well founded. Also, why would the advanced alien get any satisfaction from torturing me, since she knows i will not know i 'm being tortured.

The law protects humans (our species), probably because it's an evolutionary beneficial strategy. More specifically it protects consciousness that arises in a central nervous system (hence the abortion debate). There is no reason to assume that abstract consciousness in itself should be protected, unless it benefits the human race. Suppose the software "Consciousness v1.0" could run on your PC, would you leave your PC on forever ? Run multiple copies? Stop the people who try to update it?

Is a model of the human brain, in which the same interactions between neurons occur as in a real brain, conscious? There currently doesn't seem to be a definitive resolution to that question, and there probably never will be. However, I would argue that the difference between a "useful large scale model of neurons" (where "large scale" means an entire human brain) and "a machine that thinks" (or contains processes which think) is almost nonexistent, unless you subscribe to dualist philosophies.

I 'm an eliminativist myself, so i see no duality. It's time to demystify "consciousness". It's a word used to cover a whole range of phenomena that arise in our brain.

I think as long as we still view it as a machine, the issue with never be taken seriously.

You should watch "the measure of a man" episode of TNG...

What if the machine starts printing out "NOOO!! Don't turn me off, I'm alive in here"

Someone's going to go soft.

Even a chat-bot could be designed to plead for its "life".

Perhaps a new version of the Turing test is in order. Two computer programs, each pleading for their continued existence. The human judge/executioner is able to interact with both and has two kill switches, one for each. If one of them isn't turned off in 20 minutes, both will be turned off.

> Even a chat-bot could be designed to plead for its "life".

And people respond emotionally to those.

Also, see http://qntm.org/difference

Except instead of two computer programs, it will be one computer program and one human.

Chat Battle Royal.

What if the machine hires a lawyer to represent it?

America 2.0

So on the up side, I think turning it off would be the same as putting it to sleep. The question is do you ever turn it back on...

Reply on news.ycombinator.com