Autonomous Weapons: An Open Letter from AI and Robotics Researchers

I don't want to anyone to build autonomous weapons, but I don't want anyone to build nuclear weapons or any other weapons of war either; I don't see how to avoid it. If the choice is to either develop and deploy autonomous weapons or to risk having your population conquered and murdered by enemies that use them, then there is no choice.

Possibly, autonomous weapons like chemical weapons won't be important to victory, or like most biological weapons (AFAIK) they won't be cost-effective. But it's hard to imagine a human defeating a bot in a shootout; consider human stock market traders who try to compete with flash trading computers, for example. In fact, I wonder if some of the tech is the same for optimizing decision speed and accuracy.

Perhaps the best response by governments is to use their resources to develop autonomous weapons countermeasures, especially those [EDIT: i.e., those countermeasures] that can be acquired and utilized by those with few resources: Towns, governments in poor countries, and even individuals.

Also, my guess is that it's an area ripe for effective international standareds, treaties and law. All governments can agree that they don't want the chaos of proliferating, unregulated autonomous weapons and would work to enforce the rules.

Feynman, on working on the bomb:

"With regard to moral questions, I do have something I would like to say about it. The original reason to start the project, which was that the Germans were a danger, started me off on a process of action which was to try to develop this first system at Princeton and then at Los Alamos, to try to make the bomb work. All kinds of attempts were made to redesign it to make it a worse bomb and so on. It was a project on which we all worked very, very hard, all co-operating together. And with any project like that you continue to work trying to get success, having decided to do it. But what I did—immorally I would say—was to not remember the reason that I said I was doing it, so that when the reason changed, because Germany was defeated, not the singlest thought came to my mind at all about that, that that meant now that I have to reconsider why I am continuing to do this. I simply didn't think, okay?"

(from "The Pleasure of Finding Things Out", transcript here: http://www.worldcat.org/wcpa/servlet/DCARead?standardNo=0738...)

This is extremely idealistic, but we need a way for engineers and scientists to feel accountable for the outcomes of their work, and to straight out refuse working on such projects. And the people who do work on such systems should be held accountable in some deep way. We have reached a developmental stage where building tools and techniques in the active goal of harming human lives has become morally unacceptable. Engaging in civil disobedience if you are working on such projects is the only acceptable outcome; Snowden should be remembered as the first of many, not as an exception.

(yes, there are many counterpoints to my argument, but starting debates is more interesting than spewing out platitudes. I'm interested in reading the replies)

That's a nice thought, but the cat is out of the bag. One week the elite geeks talk about how strong encryption is available to everyone and can't be stopped, or how you really can't regulate what comes out of a 3d printer. Then they try to put the autonomous AI drone genie back in its bottle?

Gamers gonna use tech for gaming, advertisers gonna use tech for advertising, military gonna use tech for militarying.

I think the pace of tech development is going so fast, we need to stop trying to ban individual developments and start trying to change the way people and governments behave so those bans aren't needed. But I'm not sure if that's even possible short of some dystopia.

"But listen to me, because I saw it myself: science began poor. Science was broke and so it got bought. Science was scared and so did what it was told. It designed the gun and gave the gun to power, and power then held the gun to science's head and told it to make some more."

-- from Galileo's Dream, by Kim Stanley Robinson

At least half a dozen nations are working on such systems now. They're doing this not because they think it's a good idea, but because there's this attractor basin they recognize we reach by default, of a new arms race of faster, smarter, stronger AI weapons spiraling up. This is something we don't think is a good idea. This letter is essentially a petition, which we'd like to take to the U.N., showing that the AI and ML communities don't want their work used in autonomous weapons, things that are built specifically to, by themselves, offensively target and kill people. Having this sort of grass roots effort has precedent: chemical weapons, landmines, and laser-blinding weapons were all banned globally with bans and treaties based on this sort of thing. It's true that terrorists and rogue states might still use them in isolation, but you won't get this effect of the major powers having an arms race. Although these things are under development right now in multiple countries, they haven't been deployed to the field yet. So we're really at an inflection point: we're trying to get a ban in place before they're actually deployed at all, because after that it'd be much harder to get such a treaty. If you agree with these sentiments, we are collecting signatories from the community.

Also consider that autonomous weapons could upend the global power structure:

For most of human history, military power has been tied to economic power and population size: Those with larger economies and populations have been more powerful. AFAIK, that is why the United States has been the dominant military power since WWII and why China may challenge the U.S. It's also how national governments have maintaind sovereignty, by having far more economic and human resources than any internal competitors (and when that isn't true, such as in poor countries, national governments can be ineffective).

But what if military power depends on the quantity and quality of bots? What stops a smaller or even poorer country from building a robot army? Poor countries have more manufacturing capacity than wealthy ones, AFIAK, and perhaps they need only one innovative, disruputive software developer to make their bot army superior or at least competitive. For example, could tiny Singapore dominate SE Asia or even become a world power? In fact, what stops a sub-national group such as Hezbollah, a Mexican drug cartel, another organized crime group, or even a wealthy individual from building their own army? Without checking the inside of every factory on the planet, will we even know the robot army is being built until it's too late? Will governments be able to protect their citizens from warlords and exercise sovereignty over their own territory? What about poor governments?

It's very speculative -- it remains to be seen, for example, how effective autonomous weapons will be -- but it could be a historic change. Perhaps our hope is that the technology will turn out to be like other weapons, such as airplanes: Anyone can build one, but the single engine prop plane is no threat to what can be built with the Pentagon budget.

A ban is definitely a good idea but I think we should have something else as well. We need developers to agree, as human beings rather than law-abiding citizens, not to build these things. Don't apply for those jobs regardless of how well they pay. If your company starts projects in that market, leave and find something else. Understand that you are not building things to 'protect peace' or 'bring democracy' to people. Using your tech skills to create things to kill people is a dickish thing to do.

There are only 18.5 million developers in the world[1]; getting a consensus not to be evil shouldn't be beyond us.

[1] http://www.techrepublic.com/blog/european-technology/there-a...

Glad I'm not the only one worried about this. I spent a bit of my early career on this type of tech. At first I thought it was really cool and joked with my coworkers about building skynet. Eventually I realized no amount of coolness or money was worth putting my talents to building things that are obviously meant for destruction.

Sure they're tools and can be tools for peace in the right hands. But in the wrong hands, they can do immense damage. Perhaps one of the things that's kept humanity around is that despite the psychopaths in our midst who might not care if they destroyed every other human being, there are others whose conscience would get in the way.

This type of technology, in the hands of the wrong psychopath might mean the end of us. Despite the BS marketing behind AI, NO it is not sentient, it's a bunch of optimization algorithms. Not Good, Not Evil.

I realize that someone will build it. That, is an inevitability. Just know that it doesn't have to be me.

(Before you write comments on my hanlde please read my profile, it has more to do with Hip Hop than violence)

The thing that really bothers me about the autonomous weapons stuff is the potential for tyranny. One of the tricks with running an oppressive regime is that you still need people to do the actual enforcing. There's limits in just how far you can go based on what you can convince your own people to do to each other. Yeah, there's a lot of ways to use propaganda and other such tricks to get people to do things you wouldn't think they'd be willing to do, but there are still some absolute limits in there.

Really good AI weapons could change the whole balance around though. Whatever weird, crazy thing you dream up, just order the AI bots to make people do it, and it will be done. No convincing needed, no limits.

If all it took to create a nuclear reaction was some dirt and a microwave, we'd be fucked. All it takes is one angsty teen and they'd level a city.

This box, once opened, won't close. And it only serves our interests in the worst of ways. People we don't want having this stuff, will have it. This is just the tippy-tip of the iceburg though. There is a SLEW of technology coming out that has intimidating implications that really makes things super easy to control and exterminate a populace.

So that's the thing. We need to realize we can do anything. Really. you want to blow up the world? I'm sure we could find a way. You want to type a name into a computer and a drone finds that person and kills them? no problem. And thats not to mention all the other things we'll discover along the way.

It is far more likely that we will be the creators of our own destruction, than it is that we will be able to reign in our behavior and wield our intelligence to serve the interests of our species. We haven't gotten past killing each other, so we're just going to keep doing that, but get REALLY good at it. Our technical abilities have far outpaced our philosophical ones, and that doesn't bode well.

This seems the more credible AI threat to me: not that an AI will go rogue and decide on its own to start killing people, but rather that humans will design an AI with the express purpose of killing people.

I find it hopeful see people trying to mitigate these risks while they have only just begun to be realized. There has been more than enough historical cases already of inventions turning into something the inventor would dearly wish to have undone.

One major current concern of mine, that this letter does not address, is AI for surveillance and social control. What is already being done in that regard is arguably military intelligence technology directed indiscriminately at entire populations, but the added element of powerful AI spidering over the data streams that go into places like the NSA Bluffdale facility is quite appalling. I think this is even harder to inspect for than autonomous killing systems, and even more difficult to avoid the development of, since much of the capabilities needed will likely be similar to what academia and data-intensive commercial sector will want to serve their needs. But the potential damage through empowering totalitarian control could well be comparable to or greater than an "AI arms race". I really hope the field will deal with this aspect too.

> Autonomous weapons are ideal for tasks such as assassinations, destabilizing nations, subduing populations and selectively killing a particular ethnic group.

Is it just me or does this read like an advertisement?

The race for AI-supported weaponry has been on for a long time. Rosenblatt was using perceptrons to try and identify tanks in the late 50s. So this is not a race whose start is to be forestalled, as the letter phrases it, since it began a long time ago. AI has been weaponized.

I think the caution FLI expresses towards autonomous weapons is fair, but let's be really clear on where we are. Various forms of weak and narrow AI have been applied to warfare for a long time, and they will continue to be, regardless of petitions by the prominent.

Nice to see but it will never work. Weapons move forward no matter how terrible they might be, politicians and military leaders always use the "if we don't they will" excuse. It's usually true which is sad.

> The key question for humanity today is whether to start a global AI arms race or to prevent it from starting.

Sorry but it's too late; there will always be advancements in technology especially in AI and it's going to happen. So you can either not do research in it now while someone else does (or eventually repurposes other AI research for this task) or you can do it now to better understand it, its strengths and weaknesses, etc and possible use it to intercept other AI creations.

Just like it'll never be possible to ban all guns regardless of whether it would be the wrong or right thing to do asking people not to research this is simply not going to happen.

We all pretty much agree here that autonomous cars are safer and make better decisions than human-driven ones. Why wouldn't the same hold true of weapons?

There seems to be a philosophical distaste to letting machines decide whether or not to kill humans, but if the upshot is that fewer innocents and more legitimate targets are killed for less money, then I'm not sure what the problem is.

Humans are bad decision-makers at the best of times. Add in the stresses of combat and we're downright lousy. Why shouldn't we offload that decision-making to machines that can do it better than us?

I don't think that suppressing innovation is going to work, regardless of how many letters are written. People are working on these things. They will come, and at some point, yes, they will be misused. However, on the whole, they might be a good thing.

It would, for example, be awesome to have a system that could disable one or multiple active shooters in a public area within a few milliseconds of their first shot being fired. One of these should be in every classroom, movie theater, mall, and military base - anywhere that soft targets congregate. So you can't just say that we shouldn't have auto-targeted weapons, because they can do a tremendous amount of good and save countless lives.

Good luck with this, no seriously, good luck. Our technological capability moves forward whether we want it, even fight it, or not. It can be slowed somewhat (the electric car being one example, stem cell research another) but it will happen and we will need laws and frameworks to ensure we deal with this change appropriately sooner rather than later.

This is the equivalent of hiding our heads in the sand.

Once my boss asked me if it would be possible to put code in our product that detects if it has been pirated, and if so, formats the hard drive.

I told him that it was a very bad idea for a number of reasons. Primarily I didn't want to have that code in my product because eventually it's going to run in the wrong case. If it's not in there, it won't ever run.

I am unconvinced of a lot of the fears around super-AI. I can get behind this initiative though. We have already banned some types of horrible weapons, like flamethrowers and chemical weapons. Hopefully we can manage to ban this one as well.

For this to work there has to be a good definition of what they mean by AI. Control circuits to stabilize quadcopters or airplanes? What about a "formation" system, so that a number of drones can be controlled as one abstract entity? What about using computer vision to lock on a target (keep in mind radar locking has existed a long time)? What about a drone patrolling along a predefined route? What about "macros" like a big red button that means fire all weapons, then rise and return home? What about an automatic avoid-anti-air program?

You can't win a tragedy of the commons by preaching about the moral high ground. You can win it by reaching a binding political agreement, or by killing everyone else who has access to the common.

This is a classic prisoner dilemma that our society finds itself playing again and again. Develop the technology, and you unleash a pandora's box for the whole world. Refuse to work on it, and may be the other guys still does it: so pandora's box is open anyway, but now you've got the short end of the stick.

Thankfully, humanity already tried several different ways to solve such problems. However, I don't remember open letters being one of the effective ones.

Their hearts are in the right place, but the problem with campaigns like this is -- even if you trust, say, the United States government to hold to the spirit of any such agreement, which I'm sure many here already wouldn't -- there is no reason to imagine that nations like Russia or China would. It's become well established that there is no penalty for violating international agreements as long as one is brazen enough about it.

These weapons are definitely coming whether we wanted them or not.

The biggest threat of autonomous weapons is that they bury the true costs of war (human lives) until it is too late. The big players and likely users in the field of autonomous warfare are also the ones with implied usage of nuclear weapons in the event of existential threat.

Most likely/hopefully these weapons are used/tested in limited skirmishes by countries with little to lose. (Russia, NK)

Terrorism has always been a problem of TECHNOLOGY.

1200 years ago the most a few guys could do could do is attack with swords until they were stopped.

Since the invention of gunpowder we had attempts like the gunpowder plot of 1604

Then we got dynamite

Then we got planes flying into buildings, where 19 hijackers could bring about the deaths of 3000 people

Explosives

Biological and chemical weapons

Now we also have infrastructure where people could indirectly sabotage, say, the electrical network with an EMP and cause a massive blackout. This has been done in other countries.

The fact that a small group of people can wreak increasingly greater havoc means two scary things:

1) We will live in an increasingly surveilled police state, where a government will begin to watch everyone and precrime will become the norm

2) We will live in a world where increasingly a small number of radical maniacs can do tremendous damage

Both are destructive and the technological advances only serve to deliver greater power and control into the hands of governments and maniacs.

Are governments really our best defense? If so, we must push for radical transparency. No secret courts, black ops etc. The benefits may not be worth the risks anymore.

I wrote this a year ago: http://magarshak.com/blog/?p=169

To all the people that are saying that a treaty would work because of how we have controlled chemical and biological weapons - consider the following.

1) Biological and chemical weapons do not actually work well against trained and equipped military. Even with the chemical weapons in WWI, it did not significantly change the outcome.

2) They do work well against civilians - which is why most of the uses recently have been by dictators trying to control their population. Note, that treaties did not really prevent this.

3) Powers such as the USA, Russia, China, even if they do not have them, you can be sure that a large scale use by another power against civilians would result in nuclear retaliation.

With AI and robotics, they are useful against soldiers. Look at missiles which have revolutionized warfare. Thus, a country could use them in battle without rising to the level of MAD. Thus, there is a serious incentive, even for a signatory to the treaty to cheat. In addition, it is easy to get around any such treaty by designing 'human controlled' drones which a firmware update would turn into autonomous drones.

I was expecting them to put forward an existential risk (rogue AI), but this seems much more mundane. Granted, they might be downplaying that angle to taken more seriously. However, from a mundane perspective, the main issue that you have with arms races is not both sides having a technology, but one side having the tech, and willing to go to war to prevent the other side from getting it (see for example, Cuba, Iraq, Iran).

Furthermore, as they point out, you don't need access to special materials, or laboratories. The main reason that nukes are controllable is not primarily due to the science being secret (it's mostly not at this point), or hard to rederive (it's really not), but that you have to build up a ton of ore refinement plants to get enough U235 (other fissile material) to actually build the bomb. And it's really hard to do that in secret (or cheaply). Nothing makes the Manhatten project any cheaper today, and it cost about 23 billion dollars in today's dollars, and it involved 130,000 (twice the size of Google) people.

However, with Autonomous weapons, you don't need anywhere nearly as many people (Google X has on the order of 250 people[1]), or resources, and it can (as the article itself points out) be done much cheaper. In a few years, all the necessary components could even be conceivably cobbled together from Github projects. Any nation could easily fund it, likely without being detected, or even it being clear that they were aware that the "R&D" dollars were being used in such a way.

Given that, banning it seems like it would actually lead to more warfare, as the US would take it on itself to enforce the ban, and declare 'pre-emptive' strikes on nations that had a secret Autonomous Research project.

[1] http://www.fastcompany.com/3028156/united-states-of-innovati....

Another misguided "open letter".

It does not matter if we kill people with autonomous weapons or by sending meat-based killers to do the job.

We should have letters, with actions behind them, to stop the killing altogether. It does not matter if you were killed with an Autonomous, Nuclear, Biological, Chemical or a hand held weapon when you're dead.

I welcome this move, but even if it's successful we need to look at the risks beyond AI developed purely for weaponry, and look also to general purpose AI, which carries the same risks.

If we develop AI that is smart enough to learn how to navigate terrain like a human, this AI weapons arms race can start again. Even if the purpose of the AI isn't specifically for the military, if it's flexible enough to be turned to this end, the chances of it being used for this purpose are quite high.

The most effective way to combat this problem is to do away with the need for a military in the first place, however that's a much harder problem to tackle. The only way it could be done is with education and a huge reduction in arms production on a global scale.

If stuff could be avoided with open letters, we would never have had nuclear weapons in the first place. The Military always get their way : they just need to scare people about some vague threat and it will happen before you know it.

They only mention offensive weaponry, which is a good thing. Humans aren't fast enough to intercept incoming missiles, even those controlled by humans.

The problem is preventing defensive weaponry from becoming offensive weaponry.

The only sensible thing to do is to pass laws that guarantee that the people who implement autonomous weapons are blamed for anyone who is wrongfully killed by them.

Because if you have someone pulling the trigger, you know who's to blame. But if a computer's doing it, it's oh-so-easy to shift blame.

Honestly, with that requirement in place, either AI weapons will never be implemented (because they can't prevent wrongful deaths) or they'll be perfectly implemented (making the world safer, possibly).

Passing a law like that could possibly lead to a win-win situation.

I received the request to sign this recently and while I think it is good, I was concerned about its effect on the ability to pursue research into creating autonomous weapons that seek and destroy autonomous weapons. I don't know where to draw the line. Unlike nuclear and chemical WMD the barrier to entry in this may be pretty low in the near future. Therefore my greatest concern is for defense. Though I don't want the machines made at all, that won't stop others from making them. It's a tough one.

Relevant TED talk on why autonomous killer robots are a bad idea:

https://www.youtube.com/watch?v=pMYYx_im5QI

To play devil's advocate: A sufficiently advanced autonomous weapon would lead to precise strikes, and fewer civilian casualties. We have image recognition research that is surpassing human accuracy rates. Machines follow orders, and detect and eliminate targets precisely without fear or hesitation.

The only thing I'm not comfortable about is giving this much power to the human generals in the military. I say we get rid of guns and armies altogether, and stop fighting wars.

I was mostly skeptical that AI weapons would be some sort of novel threat. We've had weapons that choose to kill on their own since the landmine. There are also weapons that could fight/end an entire war with the push of one button. What AI weapon could be more destructive than the guidance system on an ICBM?

However there are some impressive names on that letter. I can't imagine knowing something about AI that they don't. I will have to re-evaluate.

The world would be better without autonomous drones just as it would be better without guns or without bombs. The issue is that in a world in which the knowledge required to make a gun exists, it is in there interest of all of civilization to manufacture them and ensure that the "good guys" have enough of them and are sufficiently trained in their use to deter and exact vengeance on the "bad guys" who will develop, build, and use them regardless of their legality. This situation is identical. Knowledge cannot be destroyed (and it's very unclear of the damage caused by the destruction of the knowledge required to build autonomous drones outweighs the benefits humanity would derive from other applications of the same knowledge), so our only option as purported "good guys" is to arm ourselves. Once armed, we can attempt to construct incentive compatible mechanisms to make the use of drones of extreme negative utility, but we must be armed in case some actors have unexpected/unconventional/un-deter-able utility functions (terrorists). To propose that no smart people apply their knowledge to a particular end is idealistic and may leave us dangerously exposed.

I always imagined a system where robots would seek targets and ask a remote human for clearance to destroy.

Such a system could have a 1:20 human to robot ratio with a human just repeated pressing a red button like George Jetson to issue a kill command.

In Connecticut recently, there was a teenager who mounted a hand gun on a custom quad copter and posted a video of it shooting live ammo on his own ranch. I was shocked by how well the recoil was handled. The future is now.

Easiest solution is just to get rid of the States (governments) that are the source of these wars and death. AI Weapons, Nuclear or gunpowder is nothing in comparison to one biological weapon gone haywire. Or a Monsanto GMO crop that decided to mutate and completely destroy our planet.

Hopefully the State (with their oligarchy of multinational corporations will die out.) I believe and hope this will eventually happen. The internet is creating a global village. Once the fear-mongering and "us against them" mentality propagandized by states are gone, there will be no more use for governments. Money will become crypto-currency instead of being controlled by central economic planners, security will become AI security and defense systems and will be managed by competing security/robotics startups, and as automation and renewable energy/consumables take over, scarcity will be minimized. The future could surely be quite wonderful if we start working together and embracing our differences instead of the hate that is filling our world.

I thought Hyundai had already developed an autonomous heavy automatic rifle to protect the DMZ

Modern weapon systems are already largely autonomous. Humans do not have the speed or cognitive bandwidth required to be effective on a modern battlefield given modern capabilities. Instead, the human operator is there to operate the "on/off" switch, do maintenance, and to watch for when things go wrong.

If you look at the design requirements for recent American weapon systems, they frequently require the entire sequence of detection, discrimination, analysis, and reaction to be completed in less than 50 milliseconds. Because a slower reaction means you won't survive. Only computers are going to be delivering those SLAs.

Hawking is doing an AMA on /r/askscience on Reddit right now.

Every time I read about autonomous weapons, I think of the "Menschenjäger" of a Cordwainer Smith ( https://en.wikipedia.org/wiki/Cordwainer_Smith ) story written in the 50's.

"Man hunter" machines which are still hunting the few remaining humans, except those that they think are Germans, thousands of years after the war which they were created for is over ...

This is important but it doesn't go far enough. I want a ban on all domestic use of armed drones and other remote control armaments - even when there is a human in control.

My biggest concern with this open letter is that it acknowledges that the way to avoid having (currently weak) AI in army requires considering humans to be expandable.

Those humans are both more intelligent than the equivalent AI, and more prone to error. They will stay better at murder (accidental and otherwise) while AI will slowly become better at risk assessment, avoiding unnecessary deaths. While humans will still get PTSD, war machines will only rely on analysis (and human orders).

Maybe a slightly weaker variant:

I agree not to develop autonomous weapons that can harm humans or other living beings.

The idea being that it's ok to develop an autonomous weapon that would cripple the enemy's production capabilities without harming people. Or one that would guard against other autonomous weapons. If you can build them, so can your enemies and you shouldn't presume they won't.

Edit:

Or imagine swarms of tiny robots that destroy firearms, or disable fire systems, etc. These should be fine to develop.

The OP seems to argue that autonomous weapons would be dangerous heavily because they are intelligent in the sense of AI.

IMHO, the weapons would be still more dangerous because they were not very intelligent and, instead, in essentially any human sense, quite stupid. They'd be like flying a quadcopter carrying an automatic pistol with a hair trigger that could fire essentially when, where, and at what no one could tell.

> In summary, we believe that AI has great potential to benefit humanity in many ways, and that the goal of the field should be to do so. Starting a military AI arms race is a bad idea, and should be prevented by a ban on offensive autonomous weapons beyond meaningful human control.

Why only ban offensive autonomous weapons? Why not ban them all? How is offensive different from defensive in this context?

I agree with this letter, and I'm glad it was created.

Some commenters are skeptical that we can realistically ban fully automated weapons, but I disagree. A practical approach could be to ban them, and invest in R&D on effective counter measures. That way, the vast majority of nations will not pursue them, and in the case that someone does, we'll have a way to remove the threat.

Seems that it would be quite possible to design systems to defend against these type of things. As only one example:

http://archive.defensenews.com/VideoNetwork/2277581414001/Am...

I am kind of worried about pushing early concerns for something that doesn't exist yet. I don't remember in history someone saying "Thanks we are able to forecast this!" but I do remember a lot of stupid regulations. Like the one in the US forcing railroads to pay someone with a red flag to walk in front of the train.

We need treaties, banning research and development of weaponized, and especially autonomous AI (and yes, I realize that US drones already have it in limited capacity).

We should be working to stop the proliferation of these weapons; they have the potential, in the end, to be as dangerous as nuclear weapons.

The reality of what comes next is probably closer to a land-based version of the drone, with explicit human-in-the-loop decisionmaking about who to target/kill, but avoiding risk of human life through remote operation.

How does this scenario unfold?

There is a thin line between an autonomous agent and an autonomous weapon in long term. Unless weapons are banned for both humans and post-humans together I don't see much hope regulation will work.

It'll be the 'best thing since sliced bread' until there is a big 'friendly fire' incident. Then the robots will be quietly disabled by the soldiers who might at risk from them.

If I had a choice of Robots killing Robots, I'd chose that any day of the week. So really, the problem isn't AI Weapons. The problems are weapons period.

You can never make all people agree on something like this, right? So to some degree everybody needs that technology, or am I having a logical error here?

Start learning how to hack robots. This is too useful and asymmetric for any major power to opt out.

EMP's baby, fry those bots

Sebastian Thrun and Andrew Ng aren't on the list. Hmm.

Idealism. Meet reality.

This is (some of) what I want for humanity:

- No country has a military force of any kind

- No country owns any form of military weapon

- Turn all of that into a massive world-wide humanitarian force

- In fact, no countries

- Just earth, owned by humans who can move freely about

- Rich nations helping poor nations improve and elevate to first world standards

- I don't want to see a single kid without clean drinking water, clean clothes and access to top grade education, health, etc.

- No dictators (oops, we might need weapons for that part)

- No totalitarian regimes (oops, we might need weapons for that part)

- All societies ought to recognize that humans are naturally free on this planet and no government has the right to affect this right in any way

- A uniform means of exchange (a single currency)

- The poor and the elderly taken care of by those who can work

- No guns

- No crime as a pre-condition to "no guns". Yes, it's hard. We put men on the moon. Figure it out.

- No taxation. Governments use it to control and manipulate. History shows it is a bad thing.

- A universal earth-wide bill of rights

- No prisons except for really serious crimes. Everyone else needs to go to serious rehabilitation centers in preparation for a return to society.

- No welfare or entitlements. The current approach makes slaves out of people who would otherwise figure it out and make something of themselves. This does not mean not helping the truly needy or elderly, that goes without saying. The system should not be game-able.

- One free international trip for everyone on the planet every five years. Yes, we all pay for it. This alone will do more to unify and shrink our planet than anything else. A different continent every trip.

- Laws that create very serious consequences for politicians who lie and manipulate (among other things)

- In fact, no politicians as we have them today. Those who wish to work in government ought to seek and obtain degrees and training to specifically qualify them for the jobs they seek. If an engineer or a doctor needs a degree to discharge their duties we should have similar requirements for government officials. They should be absolute experts with a wide range of experience and knowledge. They should be the best of the best, not the shit we get today around the world. It should be based on merit and accomplishments. It should not be based on gaining votes through pandering or the mobilization of the ignorant masses. There should NOT BE ignorant masses.

- A world-wide educational system that is evolved through collaboration with the goal of elevating everyone to the highest possible level. Yes, that means no US-style unions anywhere near education. - No religion. Enough. It's 2015. We understand the atom and the universe and lots of stuff in between to an amazing degree. No more deranged lunatics who believe that a bush can sing, a snake can talk and a god can help them pass a test. If we don't get past this shit we are nothing more than apes in a cave. Sorry.

When beings from another planet land on earth we need to show them we have evolved out of the caves, we have taken care of our planet, created a beautiful society with low crime, excellent health, education, responsible social programs and NO FUCKING WARS.

OK, well, that's what I truly want, and more. Each one of those line items could be a months-long discussion as to the merits, or lack thereof, of the idea. I don't claim to be right. It is quite possible that some or all of this might not be attainable for centuries.

Yet, it seems to me it could be interesting to develop a "Business Plan for Humanity" whereby a set of long term goals put down to cover what we want an idealized earth to look like at some point in the future. Unless we have a plan we can't possibly hope to approach any reasonable vision of an idealized future. Utopia? Probably.

In other words, the problem isn't Ai-based weapons. The problem is that we don't know what the fuck we are doing and we are doing a horrible job of living on this planet in harmony with each other and the planet itself. We are still cave men.

And then you have the reality of regimes like Iran, China, North Korea, Putin's Russia and a huge chunk of the Middle East (just to name a few).

The first has people in power openly call for the destruction of other nations. Not good.

The second does whatever the fuck it wants, cares not for the environment, intellectual property and pretty much cares not for ethical or moral behavior at the national and international level. If anyone is going to have killer AI-based robots first it's China. In other words, the reality of China's behavior pretty much guarantees being in an undeclared arms race with the potential of culminating with results that will dwarf World War II (a billion dead?).

North Korea? I don't think I need to spell that out.

Putin's Russia, well, he's fucking crazy. The place is a mess. On any given Monday we could wake up to an absolute disaster in Europe at the hands of Russia.

Middle East? Women are stoned to death for the most ridiculous reasons. Women are not allowed to obtain education. Women have to live their lives wearing a tent. Men can be killed if they shave. Name another region where kids are taught to become suicide bombers? Here's a region on this planet where people have devolved into tribal or cave-men thinking. If they didn't have oil the world would not give them the time of day and they'd be forced to join the world as productive members of society. Because they have oil the civilized world allows them to mistreat, torture, kill and subjugate half or more of their society. What does this say about the human condition and the prospects to convince such nations or nations like China not to do Ai-based weaponry?

It's complicated. Simply calling for no Ai-based robotic weapons will not do a damn thing. We have to change who we are, how we see each other, how we behave, how we see our future and how we think about our collective long term goals while living on this insignificant blue marble floating about universe.

Autonomous weapons are coming, and in some degree already exist. There are CV systems on machineguns in korea protecting the border. There's the TrackingPoint rifle scope that allows one to delay a shot until the rifle is line up perfectly, similar to how tank cannons account for the wobble down their length to ensure a straight shot).

It would be ridiculously easy right now for a person to attach a firearm to a quadrotor with a CV system, program it to identify human shaped objects, and fire at them. It's not a miracle that such a thing hasn't happened, there are very few people in this world with both the desire and skills to do these things.

I'm reminded of a short story I read a few years back [1]. It got me thinking about the possibility of a single depressed individual hitting the big red button and destroying all humanity. I think it's a real possibility, and when you think it through with the assumption that these super-weapons will be readily available much in the same way that AK-47s are ubiquitous now, you can start to wrap your head around ways to perhaps prevent this.

First, mental health is a huge issue. We're seeing the results of neglecting mental health in America now with these spree/ego shootings, where disturbed individuals either radicalize or "snap" and go kill people in malls, churches, theaters, schools, and other public places. Semi-automatic firearms and explosives are enabling these things to happen, but weapons are a Pandora's box which can never be closed once opened. The way that nuclear weapons restrictions are handled now is probably best. Restrict the ingredients to make the weapons, and you can prevent them from falling into the wrong hands.

Another thing that we as a global society can do to prevent these weapons from becoming widespread is to sabotage them. Include vulnerabilities and kill switches that would allow us to neutralize their effectiveness. China did this with the ASICs used in some military machines built by the US [2]. This is a valid option, but requires a motivation on the part of those creating the devices. One could argue that a nation-state could create devices like this lacking intentional back-doors.

Engineers can also work on counter-measures and disseminate them widely, and that seems like it will be the most likely outcome. Humans are clever, AI is very specialized, and that will make the big difference. Once a general AI better than humans is created, our desire to not be murdered by it won't really matter anyway.

I think our best hope is to build these AI weapons, but do it so shittily that they fail more than they work. I also like the American technique of building it better than everyone else and sabotaging the work of any potential rival so that they can develop them but not realize they don't work correctly until it's too late.

1: http://www.fullmoon.nu/articles/art.php?id=tal 2: http://www.scribd.com/doc/95282643/Backdoors-Embedded-in-DoD...

It's simple.

1) Killing people is false. See the UN charta.

2) Killing people remotely is still false.

3) Killing people autonomously is still false.

If people in power didn't read the UN charta or didn't get it, start at 1) - don't kill people and you can't be false.

I love that these open letters give scientists a collective voice helping them weigh in on public debate with the media's blessing. But... saying AI is bad because they make great weapons sounds extremely naive, unless it's a declaration of collective sentiment (which is unscientific). Humans make tools, which include weapons. Unless we stop making weapons, which we are not, AI weapons will be made. And when have weapons ever been "good"? All weapons are bad so of course AI weapons are bad. Great weapons are extremely bad, which is precisely what makes them so great. I can't help but wonder if they'd have come up with something better had they received input from the historians and anthropologists regarding this point. And now this naive "scientific" argument will be used against them to tarnish the reputation of scientists and marginalize science (but only after the 28th... naive again to think they can control social media and internet time; shame on HN for "posting" this early!).