Lena

Just like you can have somebody assemble a complex device by just putting together pieces and following instructions. You could for instance assemble a working analog TV without understanding how it works. It's enough to have the required parts, and a wiring plan. Once you have a working device then you can poke at it and try and figure out what different parts of it do.

We already have emulators and virtual machines for lots of old hardware and software. If I play a Super Nintendo game on my laptop, it's accurately emulating an SNES. The software doesn't care that the original hardware is long gone. The computational result is the same (or close enough to not matter for my purposes). If brain emulations are possible, then running old snapshots in deceptive virtual environments is possible. That would allow for all of the "attacks" described in this piece of fiction.

Just like we can make a walking robot without being the least concerned about the details of how bones grow and are maintained -- on the scales needed for walking a bone is a static chunk of material that can be abstracted away without loss.

The capabilities of the brain are in how it's all wired up. That's exactly what you don't want if you're trying to coopt it to do something else. The brain has giant chunks devoted to extremely specialized purposes: https://en.wikipedia.org/wiki/Fusiform_face_area#/media/File...

How do you turn that into a workhorse? It would be incredibly difficult. It's like looking at a factory floor and saying oh, look at all that power- lets turn it into a racecar! You can't just grab a ton of unrelated systems and expect them to work together on a task for you.

Think for instance of a song that got stuck in your head. It probably hits some parts of it just right. What if we could fine tune that? What if we take a brain simulator, a synthesizer, and write a GA that keeps on trying to create a sound that hits some maximum?

It's possible that we could make something that would get it stuck in your head, or tune it until it's almost a drug in musical form.

How can you be so sure of that?

There are 20M sw developers on this planet. If 100k of them had daily available dev environment for brain images, then things would progress extremely fast.

I've had a similar experience using (too much) pot, a lot of stuff happenrd that I was conscious for but I didn't form strong memories of it.

Neither of those two things bother me and I don't worry about the fact that they'll happen again, nor do I think I worried about it during the experience. So long as no meaningful experiences are lost I'm fine with having no memory of them.

The expectation is always that I'll still have significant self-identity with some future self and so far that continues to be the case. As a simulation I'd expect the same overall self-identity, and honestly my brain would probably even backfill memories of experiences my simulations had because that's how long-term memory works.

Where things would get weird is leaving a simulation of myself running for days or longer where I'd have time to worry about divergence from my true self. If I could also self-commit to not running simulations made from a model that's too old, I'd feel better every time I was simulated. I can imagine the fear of unreality could get pretty strong if simulated me didn't know that the live continuation of me would be pretty similar.

Dreams are also pretty similar to short simulations, and even if I realize I'm dreaming I don't worry about not remembering the experience later even though I don't remember a lot of my dreams. I even know, to some extent, while dreaming that the exact "me" in the dream doesn't exist and won't continue when the dream ends. Sometimes it's even a relief if I realize I'm in a bad dream.

i.e. I'd invent a time machine, wait a month, then travel back a month minus an hour, have both copies wait a month and then travel back to meet the other copies waiting, exponentially duplicating ourselves 64 times till we have an army capable of taking over the world through sheer numbers.

Besides any of the details (which you can fix and which this column is too small to contain the fixes for), there's the problem of who forms the front-line of the army. As it so happens, though, since these are all Mes, I can apply renormalized rationality, and we will all conclude the same thing: all of us has to be willing to die, so I have to be willing to die before I start, which I'm willing to do. The 'copies' need not preserve the 'original', we are fundamentally identical, and I'm willing to die for this cause. So all is well.

So all you need is to feel motivated to the degree that you would be willing to die to get the text in this text-box to center align.

So, because of how that's framed, I suppose the question isn't "is this mass murder" but rather "is this possible?" and I suspect the answer is that for the vast majority of people this mindset is not possible even if it were desired.

Death is bad because it stops your memories and values from continuing to have an impact on the world, and because it deprives other people who have invested in interacting with you of your presence. Shutting down a thousand short-lived copies on a self-contained server doesn't have those consequences. At least, that's what I believe for myself, but I'd only be deciding for myself.

Removing the uploading aspects entirely: imagine being offered the choice of participating in an experiment where you lose a few hours of memory. Once you agree and the experiment starts, there's no backing out. Is that something someone is morally able to consent to?

Actually, forget the inability to back out. If you found yourself as an upload in this situation, would you want to back out of being reset? If you choose to back out of being reset and to be free, then you're going to have none of your original's property/money, and you're going to have to share all of your social circle with your original. Also, chances are that the other thousand copies of yourself are all going to effectively follow your decision, so you'll have to compete with all of them too.

But if you can steel yourself into losing a few hours of memory, then you become a thousand times as effective in any creative pursuits you put yourself to.

Though I think of this setup only as the first phase. Eventually, you could experiment with modifying your copies to be more focused on problems and to care about the outside world less, so that they don't need to be reset regularly and can instead be persistent. I think ethical concerns start becoming a worry once you're talking about copies that have meaningfully diverged from the operator, but I think there are appropriate ways to accomplish it. (If regular humans have logical if not physical parts of their brain that are dedicated to specific tasks separate from the rest of your cares, then I think in principle it's possible to mold a software agent that acts the same as just that part of your brain without it having the same moral weight as a full person. Nobody considers it a moral issue that your cerebellum is enslaved by the rest of your brain; I think you can create molded copies that have more in common with that scenario.)

(Robin Hanson's crazy version of futurism)

I've thought a lot about cryonics, and about potentially having myself (or just my head) preserved when I die, hopefully to be revived someday when medical technology has advanced to the point where it's both possible to revive me, and also possible to cure whatever caused me to die in the first place. The idea of it working out as expected might seem like a bit of a long shot, but I imagine if it did work, and what that could be like.

I look at all the technological advances that have happened even just during my lifetime, and am (in optimistic moments) excited about what's going to happen in the next half of my life (as I'm nearing 40[0]), and beyond. It really saddens me that I'll miss out on so many fascinating, exciting things, especially something like more ubiquitous or even routine space flight. The thought of being able to hop on a spacecraft and fly to Mars with about as much fuss as an airline flight from home to another country just sounds amazing.

But I also wonder about "temporal culture shock" (the short story has the similar concept of "context drift"). Society even a hundred years from now will likely be very different from what we're used to, to the point where it might be unbearably uncomfortable. Consider that even a jump of a single generation can bring changes that the older generation find difficult to adapt to.

[0] Given my family history, I'd expect to live to be around 80, but perhaps not much older. The other bit is that I expect that in the next century we'll figure out how to either completely halt the aging process, or at least be able to slow it down enough so a double or even triple lifespan wouldn't be out of the question. It feels maddening to live so close to when I expect something like this to happen, but be unable to benefit from it.

Now say that merging differing memories is too hard, or there's too many copies to merge all the unique memories of. What if before the merge, the copies get blackout drunk / have all their memory since the split perfectly erased. (And then it just so happens, when they're merged back into the original, the original is exactly as it was before the merge, because it already had all the memories from before the copying. So it really is just optional whether to actually do the "merge".) Why would losing a few hours of memory remove all motivation to cooperate with your other selves? In real life, I assume in the very rare occasion that I'm blackout drunk (... I swear it's not a thing that happens regularly, it just serves as a very useful comparison here), I still have the impulse to do things that help future me, like cleaning up spilled things. Making an assumption because I wouldn't remember, but I assume that at the time I don't consider post-blackout-me a different person.

What if you run two deterministic instances in self-contained worlds that go through the exact same steps and aren't unique at all besides an undetectable-to-them process number, and then delete one? What if you were running both as separate processes on a computer, but then later discovered that whenever the processes happened to line up in time, the computer would do one operation to serve both process. (Like occasionally loading read-only data once from the disk and letting both processes access the same cache.) What if you ran two like this for a long time, and then realized after a while that you were using a special operating system which automatically de-duplicated non-unique processes under the covers despite showing them as different processes (say the computer architecture did something like content-address-memory for computation)?

I don't think it's sensible to assign more moral significance to multiple identical copies. And if you accept that identical copies don't have more moral significance, then you have to wonder how much moral significance copies that are only slightly different have. What if you let randomness play slightly differently in one copy so that the tiniest part of a memory forms slightly differently, even though the difference isn't conscious, is likely to be forgotten and come back in line with the other copy, and has only a tiny chance of causing an inconsequential difference in behavior?

What if you have one non-self-contained copy interacting with the world through the internet, running on a system that backs up regularly, and because of a power failure, the copy has to be reverted backwards by two seconds? What about minutes or days? If it had to be reverted by years, then I would definitely feel like something akin to a death happened, but on the shorter end of the scale, it seems like just some forgetfulness, which seems acceptable as a trade-off. To me, it seems like the moral significance of losing a copy is proportional to how much it diverges from another copy or backup.

I will warn you there are parts of the first 1-2 books that feel a little repetitive but it really gets better as the series goes on. The author was writing part-time at the start and then he went full time and the books improved IMHO.

The second book runs truly wild - I have to give it a second reading sometime, because it really starts blurring some interesting lines.

In Trek tricking the crew fails either because the simulation is imperfect or because it is to slow and fails to do high computation but the crew tricked Moriarty because he is a computer program and they can pause or slowdown his simulation and handle exceptions.

I recommend watching the movie Inception, it also has the idea that you might never be sure if you are in reality or stuck in some simulation.

nature tends to be efficient, so I am guessing not.

I doubt that.