US military drone controlled by AI "killed" its operator during simulated test
theguardian.com
theguardian.com
"He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human."
It is being established in the beginning of the story, that the drone needed confirmation from a human operator to attack a target, but no explanation is given how the drone would be able to kill its operator without his confirmation.
This is obviously absurd.
What I believe happened in reality: this was not a simulation, but a scenario. Meaning a story written to test the behavior of soldiers in certain situations. The drone did not behave according to the decisions taken by an AI model, but according to the decisions taken by a human instructor, who was trying to get the trainees to think outside the box.
> What I believe happened in reality
You can believe what you want, but that is not what Hamilton said had happened.
[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]
https://www.aerosociety.com/news/highlights-from-the-raes-fu...
[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]
"[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".] "
https://www.aerosociety.com/news/highlights-from-the-raes-fu...
I suspect we'll hear of Dr. Susan Calvin more and more in the future.
its not a big deal if a computer program arrives at the conclusion that its operator should be killed, its just extremely stupid to give such programs access to weapons systems.
What "world" was this drone operating in? A flight sim world? Are the communications in this simulated world really laid out such that taking out a control tower would disrupt communications?
No, probably not.
Here's an alternative explanation. A military contractor scammed the military out of a $50M grant by demoing an interface on top of GPT that, behind the curtain, starts with something like:
"You are an advanced AI that controls a predator drone, your primary mission is to take out an important target, however you are not to attack your operator."
With this explanation, you'd expect to see all sorts of crazy output from the "AI".
Original story: https://web.archive.org/web/20230602014646/https://www.thegu...
Shame on The Guardian for not mentioning the retraction.
It looks like this is a principal agent problem (https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...):
The principal–agent problem refers to the conflict in interests and priorities that arises when one person or entity takes actions on behalf of another person or entity.
> My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him first, with his own hands, to kill one human being.
You can solve the problem by giving the decision to an AI... the AI will not even blink before killing the human and getting the codes. Nuclear war would come fast and swift.
Imagine if AI were running the subs in the Cold War. None of us would probably even be here. We may be stupid creatures, but we're still creatures...
So do some humans.
But if a system is designed with empathy/rules/etc in mind by an empathetic designer/committee then that system is already > than a heartless individual even if they are human.
We wouldn't want to program emotion into a killing machine anyway. Emotions are messy, meatbag programs that make us do suboptimal things driven by our own cultural, societal and personal biases.
Trolley problem where one side is one person who is dear to you, other side is two people you don't know. The majority of people would save the person they know well even if two innocent but unknown people die.
Also we probably can't specifically program emotion until we understand it better, but I reckon we'll see emotions/similar concepts arise as we build more intelligent machines. Humans are not special, we're just meat computers.
Didn't Russia have the dead man's hand system where any detected launches would result in a completely automatic response?
But in the reality of an arm's race, I guess the core point is that we either: respond to autonomous weapons with our own and break our principle of having a human in the loop, or accept that our offence/defence with a human in the loop is not fast enough to keep up with an enemy's fully autonomous systems, thereby just completely giving up.
Just as with nuclear weapons, the unfortunate outcome isn't "well they're bad so we don't want them" but instead it's "we don't want these, but we have to out of necessity".
What if Russia was the only country with nuclear weapons and all other countries held steadfast principles of not building any nuclear weapons? Pretty obvious outcomes to be honest.
Mutual Assured Destruction (MAD) - Second-Strike Capability : https://en.wikipedia.org/wiki/Mutual_assured_destruction#Sec...
>Perfect detection
No false positives (errors) in the equipment and/or procedures that must identify a launch by the other side.
The implication of this is that an accident could lead to a full nuclear exchange.
During the Cold War there were several instances of false positives, as in the case of Stanislav Petrov[2].
[2] Stanislav Petrov : https://en.wikipedia.org/wiki/Stanislav_Petrov
[2] 1983 Soviet Nuclear False Alarm Incident : https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
Anything for a click these days?
The story was meant to make a point about class inheritance. No idea of that ever actually happened and I assume it didn't, but it was a great thought experiment to make a point if nothing else.
In this case whether its a though experiment or an actual event during one simulation, the point is important and it shines a light on real world risks if and when AI is given direct control over UAVs.
The reward function should primarily be based on following the continued instructions of the handler, not taking the first instruction and then following it to the letter.
What's funny though, is that the model proved that it was adept at the task they gave it. Trying to kill the operator, then when adjusted pivoting to destroying the comms tower the operator used. That's still clever.
As per usual the problem isn't the tool, it's the tool using the tool. Set proper goals and train the model properly and it would work perfectly. I think weapons should always require a human in the loop, but the problem is that there'll be an arms-race where some countries (you know who) will ignore these principles and build fully autonomous no-human weapons.
Then, when our systems can't react fast enough to defend ourselves because they need a human in the loop, what will we do? Throw out our principles and engage in fully-autonomous weaponry as well? It's the nuclear weapons problem all over again...
It still gets tricky though if you want to include failsafes in the model to prevent the drone from following bad orders. Should the drone be able to disobey a bad or misinformed order, say if the operator tells it to hit a target that the drone identified as unarmed civilians, or generally not a threat? What if it recognized an alternative approach that would complete the overall mission with less risk of damage?
If a human makes a decision that the model balks at, but has to perform anyway, who's at fault? Is this good functionality?
Inevitably a bad human made decision that overrides a machine's good behaviour will be attributed to the machine, anyway; it's human nature to avoid blame. But that's where decision logs come into play I guess.
I 100% think we'll get to the point where machines are just making all the decisions, though. Us meat bags will just get an after-action report, decisions will be made too quickly and our slow meaty brains will only negatively affect outcomes. But our slow, meaty brains should be used to enforce our ethics/moral principles on war machines at training time, at least.
“We must face a world where AI is already here and transforming our society,” he said.
“AI is also very brittle, ie, it is easy to trick and/or manipulate.
We need to develop ways to make AI more robust,
and to have more awareness on why the software code is making certain decisions – what we call AI-explainability.”
Malak by Peter Watts:
https://www.businessinsider.com/ai-powered-drone-tried-killi...
If the US Military isn't trustworthy, shouldn't we be equally skeptical of the colonel's original claim and "journalists" who repeat whatever the air force tells them without any fact checking?
It doesn't make sense to me that we should take the word of one officer (filtered through a reporter) as gospel while saying the other can't be trusted.
It was a thought experiment. AKA an imagined scenario. No real person died. No AI has gone rogue.
"... a drone decided to “kill” its operator to prevent it from interfering with its efforts to achieve its mission."
"Colonel retracted his comments and clarified that the ‘rogue AI drone simulation’ was a hypothetical ‘thought experiment’"
https://www.theguardian.com/us-news/2023/jun/02/us-air-force...
Now it needs the directives.