What's wrong with wanting a “human in the loop”?
warontherocks.com
warontherocks.com
Axiomatically: Any system, no matter how well guarded, can and eventually will be taken by an enemy and used against you.
That's why we;
Don't take a knife to fight if you're not trained to use it.
Don't build offensive cyberweapons your civilian population are not equipt to defend against (looking at you US Cyber Command).
[1] https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incid...
[2] https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940)
SpaceX's participation[2] is particularly concerning (they just built and launched four "secret" satellites for it last week) and employees are now complaining about it on Glassdoor[3]
Elon talks a lot about AI / Skynet, and it appears there is a serious attempt to construct it.
[0] https://crsreports.congress.gov/product/pdf/IF/IF11623 [1] https://en.m.wikipedia.org/wiki/Michael_D._Griffin#Career [2] https://mobile.twitter.com/planet4589/status/153896005141464... [3] https://www.glassdoor.com/Reviews/Employee-Review-SpaceX-RVW...
We've ran spy satellites since the 60s, and missile detection satellites after.. Most leading space-faring nations have also tested satellites that are technically dual use - one man's repair mission is another woman's satellite destruction capability. As well, of course, the Anti-satellite weaponry...The DoD is a big user of SATCOM, and GPS of course, is all about military targeting and coordinates.
Every weather satellite likewise is also an incredibly useful data provider for planning air operation campaigns, and an integral part of battle planning. Mix that with terrestrial and in-situ data sources, both classified, and unclassified, and you have understanding of the potential great levels of detail useful for the military.
I've observed that Mike (and other supporters of this) are very religious. I suspect they are OK gambling with our collective future. Maybe they believe God will intervene or something. Well some of us don't want to play that game.
It goes back to Sun Tzu, and up to the present day. One of its most recent manifestations is the US equipment captured by the Taliban.
The most interesting part of the RQ-170 story however is how they managed to track it accurately enough for GPS spoofing to begin with - you need a track accurate to a few dozen meters to do that, and it's supposed to be super stealthy even beyond the stealthiest fighter jets making that impossible.
The examples of the systems seems completely out of whack though.
The active participant advocates are clearly taking about things like air to ground hunter killer missions on ambiguous targets. For example, a civilian vehicle converted to technical vs a plumber hauling some piping. A farmer with a weed killer in a ditch by the road vs an insurgent constructing an ambush. These are clearly beyond the realm of what the other side is talking about - easily identifying shit like russian airforce craft with no friendly transponders and armed active radar payloads under its wings.
I don't understand the conversation that is trying to take place. Clearly they are too different to even consider. Is the conversation supposed to be hypothetical enough that we can pretend we can act in the first case, as we can in the second? Then the conversation seems pointless. Of course you can, if you accept the premise of the second in the first place.
As an aside, the section that goes like:
> exerting over the system when the system has been fine-tuned to make it easier for the human to do less, and for the operator to accept the machine’s judgment.
Is just incredibly bad faith nonsense. Or extremely ignorant. It's hard to tell which.
I think that’s the kind of case where you run into these questions about a “human in the loop” — eg, do we need a human to confirm target identification or can the robot automatically engage anyone who points a “firearm” at a soldier, even if that occasionally shoots farmers holding tools oddly?
CROWS are remote turrets with cameras and automated aiming systems; all we’d need to do is add target recognition.
But is that ethnical?
I really don't know what you expect in ways of an answer.
Of course, they were equally confident logic and expert systems could build an AI and despite none of that working out at all, it doesn’t seem to have stopped them.
The expert-system/GOFAI people are mostly not the same people.
I think last week someone on twitter told me an AI could be bad because it'd "kill us all by releasing viruses into the atmosphere", but if you had a baby it could also grow up to do that, and (correctly) nobody worries about this. Not clear why these are supposed to be different.
More importantly, it mostly relies on thinking AGIs behave like computer programs in that 1. they don't get bored 2. they can be copied or copy themselves. Is that necessarily true? If someone copied you could you take over the world easier? Maybe you'd just fight over who gets the bank account.
But I am not sure if it helps, when people talk about it, who do not have a clue at all. Which is sadly most people. (computers are a black box to them)
This will lead to situations, when activists and politicians lobby for automatic AI image control to fight child porn, not understanding false positives and that this means that lots of legitimate and very private pictures would be send of away for some human to check.
As it is I see AI researchers engaging in a curious doublethink: both assuming that someone else is thinking about the ethical questions, and at the same time whenever someone raises an ethical critique, dismissing it as technically unsophisticated.
One must remember that ethics is philosophy that has only so much traction where it touches necessity and survival.
>Is just incredibly bad faith nonsense. Or extremely ignorant. It's hard to tell which.
When you talk to the military you have to remember that many levels of military are regarded as drones that go through motions and that much of what goes on is assumed to be defined without critical thinking. When things deviate, they ask for directions from higher levels where critical thinking occurs. This is different than assumptions you typically make with end users, you assume your system is being used by people who actively think, question, and have varied needs.
In this light that argument makes a bit more sense, these are just processes that you're refining to make the process more fluid--"the human is a necessary evil because we lack the technology to do these and need meatsuits to execute" sort of thought process. Point there and shoot here, etc. When you think of much military action in this context, that people follow orders, execute predefined processes and don't think, this argument makes more sense... if I have biological robots why not replace them with faster non-biologicial robots? I completely disagree with this ideology but it's pretty deeply seated in the military from what I understand.
The issue is people arguing for HITL (human-in-the-loop) don't want drones who don't think and they assume they're able to intervene and say "no" when something absurd, unethical, or ridiculous is thrown in the mix. They want them to consider when someone says, fire a missile here, what are the implications of that to some degree. That if on their radar they see a playground full of children they might reject that order or improper information, give feedback and even ask for more confirmation.
I find the majority of the arguments presented in this article pretty weak, but don't feel like dissecting them all. The first does look at life through a Tayloristic perspective of efficiency above all else which I think is a fundamental problem humanity is facing everywhere, not just the military: the questioning of Taylorism and utility theory perspectives that dominant the world today, instead seeing a need for balancing the needs of humanity and considering things humanistically in a world increasingly ruled by the former.
This is the exact opposite to modern Western military doctrine. All levels are expected to exercise critical thinking when facing questions rife with nuance and subtlety. These range from how to employ complex weapons systems in tactical environments to abstract legal questions such as "Is this order lawful?".
At all levels, from officer to NCO to enlisted, troops are expected to employ a high level of flexibility and creativity in carrying out their orders, to be able to step up to cover the responsibilities of higher ranks if necessary, and to hold a high level of situational awareness.
Modern professional soldiers are not drones.
Frankly, I'd go so far as to say that ignorance displayed in your comment here should all but disqualify you from holding an opinion on military matters.
A British Army soldier, literate but probably with nothing beyond high school education, is expected to learn tactics needed to achieve strategic goals without babysitting. "Take that building" is a goal, working as a group, advancing under cover and flanking the enemy is tactics. Modern infantry work mostly in small units, a handful of people with a shared purpose, they have an NCO leader to make decisions but they are individually expected to understand the tactics. The officer commanding dozens or hundreds of troops remotely can't babysit them all even if they both wanted to and had the information to attempt it which they don't.
The preference to not use humans is sentiment. Drones, robot guns, and so on don't leave grieving friends and relatives, they're just machines, and will be replaced. Sentiment is a massive political problem, the American people will pay for a $100Bn/ year war without flinching, hey it's full employment at the weapons factory - but when the planes full of coffins come home that's a voters' kid in each box.
Imagine launching your autonomous weapons, having them scan the area for enemy targets, only see you, conclude that you would not have launched if there were no enemies so obviously the one and only target it sees must be an enemy, and turns around and kills you.
After that happens just a few times, the other soldiers would revolt and refuse to use the weapon system anymore. That is the sort of thing that motivates having humans in the loop, even if on average there would be greater safety without humans in the loop.
Doesn't have to be "safe" / "not dangerous" (a lot of weapons systems are very dangerous in different ways) but it should be predictable, using some reasonably simple mental model.
"Refusing to deploy" might not be a sensible option if your enemy is willing to deploy.
It seems safe to predict that human in the loop systems will be quickly overwhelmed by fully autonomous systems. The tradeoff may end up being having to balance the risk of getting killed by your own systems versus being killed by your enemy's systems, because yours were too "tame".
In the AI ethics arena, this is really one of the best arguments I've heard for the military to pursue and have automated systems in place. US leadership may agree and see all the issues with using killer robots but other countries are developing the same technology and may have less ethical concern and considerations then you do. They may be willing to let these systems fail and kill innocent bystanders or even their own soldiers. They may even look at these and consider it a necessary loss to move forward and succeed in their goals. As such, if you're not willing to make such compromises as well, unless your approach ultimately achieves the goal or you have other techniques to mitigate these corner cutting strategies, you too must cave.
If China decides to engage in conflict and releases killer robots and we don't have killer robots, can we contend? If we can't, we might have to engage in the race to the bottom that is competition unless we have an alternative mitigation strategy that doesn't require us to compromise our ethics.
I imagine this is the dilemma scientists faced during WWII when developing nuclear weapons. Part of the argument that lead to nuclear weapons was that Germany was already working on the process. This resulted in other countries engaging and ultimately building successful weapons where Germany failed. Ultimately, we now live in a world where lots of people have nuclear weapons... and overall it's been the proverbial Mexican standoff with nuclear deterrence strategies.
Is there an AI deterrence strategy? Nuclear worked because it lead to the case of a global collapse in terms of what nuclear war would bring. Killer robots that aren't sentient don't seem to have quite the same threat and seem like militaries could utilize them as they fall in an ethical grey area globally and have far less risk of ending humanity on a global scale unless we start talking about sentient terminator killer robot level uprising which is not even remotely where AI is currently or in the foreseeable future.
All of these systems are going to be as classified as possible for any country developing them, so we’d only find out once on or more of the following become true;
- it’s several decades after they’re out of date
- a politician needs a major scary saber to rattle for some reason
- they get used in a shooting war where the public can see them.
Neither of which favor us knowing current capabilities of any of the major players right now, except MAYBE Russia.
For many black box systems, the average performance may be comparable or better than humans, but when errors occur, they should fail predictably.
If systems fail unpredictably (ie unbounded in the types of errors it makes), then they can never be deployed in the field without humans in the loop.
This is why self driving cars can, at the same time, be on average safer but still not be accepted by public.
They're not safer, though.
or more realistically, your autonomous version of an aerovironment switchblade launches, has image recognition/onboard trained neural network that can identify a russian spec T-72 tank or BMP or similar, doesn't see any, loiters for a while and then self destructs.
The moon came over the horizon... then we BLEW IT UP, which is why we only have 1 moon today
The reason we need humans is for accountability. You can't court marshal a machine. Firing the operator doesn't fix anything or prevent it from happening again (next operator will be just as fallible), it just satisfies some primal part of our brains that needs to blame someone when something bad happens.
We tolerate it because there aren’t generally a lot of civilians or even friendly fire risks in submarine engagement environments.
If autonomous drones are lethal for days, weeks or even months, should we consider them a war crime?
Plenty of people die decades later from chemical and physical effects of various weapons deployed all the time (not even counting the depleted uranium issues in Iraq). White phosphorous is just one such example.
Any white phosphorus left in the environment is going to oxidize within a few days into phosphoric acid (the tart flavor of Coca-Cola), and that will get neutralized into phosphate fertilizer pretty quickly, unless it's in some kind of deadly low-pH environment like acid mine drainage.
Terrible in the moment, but except for the scars, not likely a problem once someone heals up.
A loitering drone that can be switched off at any time by its operator is a loitering drone that contains a vulnerability that enables it to be turned off at any time by an enemy.
There is some doomsday logic in deployment of ‘fire and forget’ weapons systems. ‘Once this thing is turned on even I can’t stop it’.
What if the training set just didn’t include certain cases?
I think you mixed up a couple of stories.
There was an incident in 1983 where the Soviet early warning system picked up a reflection off a satellite and high-altitude clouds and interpreted it as a launch of 5 US ICBMs. The "proper" reaction to detecting a US launch would be to counterstrike immediately, but the operator Stanislav Petrov has recognized it as an error - he correctly guessed that if push came to shove, Americans would launch a couple more missiles than just 5.
The submarine incident was in 1962 during the Cuban missile crisis. As it was ending, a US destroyer dropped a couple of small signalling depth charges above a Soviet submarine. The submarine did not have ballistic missiles, but it had torpedoes with nuclear warheads. Those weapons required usually permission from both the captain and the political officer - and both agreed that World War 3 has just started and they should engage. Thankfully, the commander of the submarine flotilla Vasily Arkhipov was aboard and in this case also his permission was needed. He instead opted to surface the ship and contact Moscow for orders.
There was no AI present during any of this.
I don't believe birds possess the required characteristics to be identified as an ICBM launch.
Nonsense; this is like saying "we changed the wheel on the bike so that the rider didn't fall off it so often which makes us question if they can ride the bike". If the interface is being changed to better deceive a user then that of course is problematic, but if it is being changed to allow the user to make accurate decisions then that is the function of the system.
>But now imagine a system similar to HARPY, a “fire-and-forget” system that targets enemy combatants in an area or enemy ships (to use an example that minimizes the collateral risk). Proponents of meaningful human control do not believe it should ever be left up to a machine to target humans.
Strawman - that is not what "Proponents of meaningful human control" believe, I am sure you can find some people who say that, but then they will also say that they don't believe that a bullet should be fired or a bomb dropped. Machines target humans all the time in the sense that unintended collateral damage occurs whenever bombs are dropped. The humans doing it must accept responsibility for this; that is the same for more sophisticated processes - the humans remain on the hook for unleashing the device. If the device decides to slaughter a primary school class that's on you in the same way as it is if you target a school yourself.
>I think there are good reasons to question this. Specifically, I am not persuaded that there is a significant moral difference between an operator identifying a single human target based on some data and a human operator drawing a box and defining targets within that box based on equally good data.
Which shows that you, dear writer, should never be allowed to decide a god damn thing. There is a chasm of uncertainty between the two scenarios, and a chasm of indirection.
>whether in cases when the system works better or as well without a human operator, we have moral reasons to place a human in the loop.
Because we live in an open world and the judgement of performance is post hoc, you can't know that it will work until it's done its thing. The judgement has to be when the act is commissioned because the consequences of the act cannot be reversed.
The biggest reason why human-in-the-loop is necessary is that humans scale better in unique complexity situations than fixed-design machines ever an be.
Often you actually want to slow down a system specifically to assure a human can judge. Fully automating assures that the small or large errors will grow far too fast - we see this with social media - that's literally the problem with Twitter et al. - they operate too fast to be safe or effective.
It's better to slow things down if it results in correct, moral or fair operation. There is nothing magical or essential or better about maximum speed blindly.
The additional big reason is that machines are incapable of morality nor of facing punishment - you need a human to "have skin in the game" to create any force to assure correct behavior or operation. Being afraid of jail or financial ruin is essential to this. A machine or corporation (NOT a person) never can and never will.
One data point is drones: currently, like most precision weapons, the precision is nearly 100% - that is, if you point the weapon at a target, the target will be destroyed nearly 100% of the time. The problem then is how you target the weapon - that's where all the "errors" are incurred. This is how you get Afghan wedding parties killed off. It's the intelligence errors and rules of engagement that are actually what kills.
Another data point that should be horrifying: the US military is currently working on neuromorphic AI chips. The specific intention is to make early warning satellites capable of identifying targets and triggering a nuclear "response/attack" without a human in the loop.
Literal Sky Net. This is intelligence and targeting being automated and humans being pulled out of the loop.
Imagine you get into a fistfight with a guy. He's bigger than you, he's got a loaded 9-millimeter pistol, and he's a good marksman. But his reaction time is 200 times yours --- say, about 50 seconds, if you're a regular human. That is, all his reactions are reacting to what you did 50 seconds ago, at best. Who do you think is going to win? Unless he's inside a tank and you're outside, you will. Even in that case, I think you've got a fighting chance.
You argue that it's good for armies to "have skin in the game" in the sense of being vulnerable to jail or financial ruin. And, from a moral point of view, it probably is good: armies that are exposed to retaliation will generally be less willing to do things that provoke retaliation, many of which are immoral. However, the things that provoke retaliation are also the things that win wars. Making your army more vulnerable may reduce war crimes, but it is not a recipe for victory.
The logic of warfare is inimical to our survival as a species, which is why our best and brightest have been calling for a pacifist world order since even before World War II. Nuclear warfare were a nice early warning, but they're children's toys compared to precision-guided munitions backed up by pervasive surveillance.
Obviously that can't apply to all situations in the same way, but I thought it was an interesting viewpoint at least. I can't remember the article or the speaker though.
Or, if you protest that "the AI did it!" - then you have to march across a field guarded by a bunch of "smart" weapon systems. The justice system is not responsible if any of those AI's malfunction and kill you.
Shall we play a game?
"Mr. McKittrick, after very careful consideration, sir, I have come to the conclusion that your new defense system...sucks."
This always managed to lighten the mood.
> At the recent conference on the ethics of AI-enabled weapons systems at the U.S. Naval Academy, well over half the talks discussed meaningful human control of AI to some extent.
But, but AI is sentient right?! What business is it of ours to try to exercise control over AI, with its many and various superior attributes? lol.
> If you work among the AI ethics community, and especially among those working on AI ethics and governance for the military, you are hard-pressed to find an article or enter a room without stumbling on someone literally or metaphorically slamming their fist on the table while exalting the importance of human control over AI and especially AI-enabled weapons.
AI ethics is only there to justify the unjustifiable. They do not have my ethics that's for sure, nor any normal person's. Its a bad joke that allows a corporation to say - 'but it has been subject to an ethical review!' - and the political class can allow themselves to be bamboozled and provide their rubber stamp.
> the ethics of AI-enabled weapons
... the ethical implications fall on those coding the dystopia.
War is immoral, pretending that wars are real is immoral, handing over decision of life and death to a machine is immoral - regardless of how efficient the machine may be. So, any of those people involved in that circle are immoral. Soldiers may think they are working to defend their 'country' - but 'country' is a fiction and 'defence' is an inversion. People act this way for money.. at least that reason makes sense - but there is nothing moral about it. There are no ethics to be found there.
Meaning is collaborative. It doesn’t matter if you’re a nihilist and want to declare your country doesn’t exist; the guy invading you makes it real.
You can be a quisling, but defecting to the invader seems to admit they exist, at least.
Do you think that 'Department of Defense' is a misnomer? Howabout 'military aid'?
Are these the sorts of linguistic meanings we are meant to collaborate on? I'm no collaborator, you know..
On the out of control thing, for me, ai is just a machine (a special machine, but not sentient as we are sometimes told) and will do whatever is most successful, as per the criteria it is given. If it is capable and can kill people, and that is stated as its goal, I'm sure it would do so. Its just a machine, it has no moral compass.
I don't buy the hype that it would be able to choose its own goals - it can never be conscious, even if it can do a passable impression.
What does happen is that the (moral) responsibility for ai deaths is shifted onto those who train and code it. Developers might not think of themselves as killers (like soldiers do), but in working on these projects that is what they become.
When I point out the moral repercussions of coding, eg here and many other examples, eg privacy erosion, gameification, people don't like it. They want to take the money but not the responsibility or culpability their actions. They will pretend that they can defer their morality to an 'ethics committee'.
So these sorts of articles are just a form of distracting entertainment, that do nothing, fail to talk about the important parts of life, and ultimately facilitate the creation of an even greater dystopia for ourselves and future generations.
For AI-in-the-loop, similarly for a long time I have framed that as an extension of heuristic-based-control-system development. I suppose implicitly that puts some or all of the onus on the developers.
But, pretending that you don't know what you do know, that you don't realise you are doing something that will negatively impact others, esp when it is easy to find these things out, won't wash. Remaining ignorant is a choice after all.
I suspect that we will judge ourselves. If you find yourself wanting, there's time to change.
> For AI-in-the-loop, similarly for a long time I have framed that as an extension of heuristic-based-control-system development. I suppose implicitly that puts some or all of the onus on the developers.
Well, yes - AI doesn't come out of nowhere, it didn't invent and code itself. At least, I don't think that's how it occurred :)
People have choices about what they do. They can choose to believe they live in a materialistic, amoral reality - as they are taught at school - and then pretend that they are 'good' with no culpability because they are doing what they are told (coding dystopia). Or they can try to become individuals and try to uncover the nature of reality for themselves and according to their principles and truth.