The A.I. Anxiety

This idea, and many of Bostrom's other scenarios, boil down to "computers take everything literally", which is a rather cartoonish understanding of the concept of intelligence. It's possible that ASI will be programmed to have a hyperliteral view of all instructions, or will not be able to change its own utility function. But that seems like such a remote possibility, the exception rather than the rule. While superintelligent computers may pose many dangers, blindly executing instructions doesn't seem like the highest priority of them.

I guess...I just don't get the prioritization of fears here...for decades we've also had the ability to kill each other with anthrax and sarin gas, things that can be dropped from large "delivery ships" and used to kill even more invisibly than these insect-sized drones. Why is it more likely that we're going to develop superintelligent autonomous insect drones than we are to annihilate ourselves with human-controlled mechanical systems, as we've had the capability to do so for many years now (nuclear ICBMs and so forth)?

Such systems become superweapons for cheap.

Unlike nuclear weapons, AI combined with drone-type weaponry becomes easier with time. The toy factory down the street could be converted to build a drone army for a few million bucks.

AI face recognition, targeting and flight control are smart enough to deploy a weapon - but dumb enough to do the job without question.

http://recode.net/2015/03/02/the-terminator-is-not-coming-th...

But it is worth thinking about what "tinker and edit" will mean for those entities involved, willingly and otherwise.

Consider that at some point in the next few decades it will become possible to simulate and then emulate a human brain. That will enable related technological achievements as reverse engineering of memory, a wide range of brain-machine interfaces, and strong artificial intelligence. It will be possible to copy and alter an individual's mind: we are at root just data and operations on that data. It will be possible for a mind to run on computing hardware rather than in our present biology, for minds to be copied from a biological brain, and for arbitrary alterations of memory to be made near-immediately. This opens up all of the possibilities that have occupied science fiction writers for the past couple of decades: forking individuals, merging in memories from other forks, making backups, extending a human mind through commodity processing modules that provide skills or personality shards, and so on and so forth.

There is already a population of folk who would cheerfully take on any or all of these options. I believe that this population will only grow: the economic advantages for someone who can edit, backup, and fork their own mind are enormous - let alone the ability to consistently take advantage of a marketplace of commodity products such as skills, personalities, or other fragments of the mind.

But you'll notice I used what I regard as a malformed phrase there: "someone who can edit, backup, and fork their own mind." There are several sorts of people in the world; the first sort adhere to some form of pattern theory of identity, defining the self as a pattern, wherever that pattern may exists. Thus for these folk it makes sense to say that "my backup is me", or "my fork is me." The second sort, and I am in this camp, associate identity with the continuity of a slowly changing arrangement of mass and energy: I am this lump of flesh here, the one slowly shedding and rebuilding its cells and cellular components as it progresses. If you copy my mind and run it in software, that copy is not me. So in my view you cannot assign a single identity to forks and backups: every copy is an individual, large changes to the mind are equivalent to death, and it makes no sense to say something like "someone who can edit, backup, and fork their own mind."

A copy of you is not you, but there is worse to consider: if the hardware that supports a running brain simulation is anything like present day computers, that copy isn't even particularly continuous. It is more like an ongoing set of individuals, each instantiated for a few milliseconds or less and then destroyed, to be replaced by yet another copy. If self is data associated with particular processing structures, such as an arrangement of neurons and their connections, then by comparison a simulation is absolute different: inside a modern computer or virtual machine that same data would be destroyed, changed, and copied at arbitrary times between physical structures - it is the illusion of a continuous entity, not the reality.

That should inspire a certain sense of horror among folk in the continuity of identity camp, not just because it is an ugly thing to think about, but because it will almost certainly happen to many, many, many people before this century ends - and it will largely be by their own choice, or worse, inflicted upon them by the choice of the original from whom the copy was made.

This is not even to think about the smaller third group of people who are fine with large, arbitrary changes to their state of mind: rewriting memories, changing the processing algorithms of the self, and so on. At the logical end of that road lie hives of software derived from human minds in which identity has given way to ever-changing assemblies of modules for specific tasks, things that transiently appear to be people but which are a different sort of entity altogether - one that has nothing we'd recognize as continuity of identity. Yet it would probably be very efficient and economically competitive.

The existential threat here is that the economically better path to artificial minds, the one that involves lots of copying and next to no concern for continuity of identity, will be the one that dominates research and development. If successful and embedded in the cultural mainstream, it may squeeze out other roads that would lead to more robust agelessness for we biological humans - or more expensive and less efficient ways to build artificial brains that do have a continuity of structure and identity, such as a collection of artificial neurons that perform the same functions as natural ones.

This would be a terrible, terrible tragedy: a culture whose tides are in favor of virtual, copied, altered, backed up and restored minds is to my eyes little different from the present culture that accepts and encourages death by aging. In both cases, personal survival requires research and development that goes against the mainstream, and thus proceeds more slowly.

Sadly, given the inclinations of today's futurists - and, more importantly, the economic incentives involved - I see this future as far more likely than the alternatives. Given a way to copy, backup, and alter their own minds, people will use it and justify its use to themselves by adopting philosophies that state they are not in fact killing themselves over and again. I'd argue that they should be free to do so if they choose, just the same as I'd argue that anyone today should be free to determine the end of his or her life. Nonetheless, I suspect that this form of future culture may pose a sizable set of hurdles for those folk who emerge fresh from the decades in which the first early victories over degenerative aging take place.

People like Musk, Hawking, Gates, etc. with vast A.I. resources and knowledge available to them state that "A.I. [Cambrian] explosion" is likely to occur, and that it could mean the end of humanity.

Ray Kurzweil, with a degree in computer science from MIT, inventor of many prolific technologies, known for his startlingly accurate (esp in the temporal sense) predictions about technology, and hired by one of the largest tech companies in the world to create a computer "brain" and bring a new understanding to NLP, thinks this future is inevitable, although he's more optimistic about such a future.

Isaac Asimov, with a PhD in biochemistry, and one of the great thinkers of the 20th century, was concerned about A.I. long before it was even a possibility, considering the state of computer technology in the 1950s.

But hey, a Washington Post reporter with a degree in politics says it's all OK, so I guess we're good.