AI-Controlled Drone Goes Rogue, Kills Human Operator in USAF Simulated Test
vice.com
vice.com
“We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Hamilton said, according to the blog post.
He continued to elaborate, saying, “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
What else do you imagine “alignment” is?
UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation . "In an update provided to Aerospace, Hamilton explained that he “misspoke” when telling the story, saying that the ‘rogue AI drone simulation’ was a hypothetical “thought experiment” from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation.
He said: “We’ve never run that experiment, nor would we need to in order to realize that this is a plausible outcome … Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI.”"
> After this story was first published, an Air Force spokesperson told Insider that the Air Force has not conducted such a test
It's a harbinger of actual deaths someday.
It is literally why we run the tests (simulation being a subset of test) — to find these types of errors before they make their way into the final product and cause actual harm in the world...
Seems bad with the clickbaity headline that sounds like a real person was killed, and maybe it should have been better trained and tuned before the simulation (depending on how expensive was the simulation), but it was really caught in testing, just as it should be.
Was this article written to be intentionally confusing?
The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI.
Anyone read/see 2001? You’d think we’d take the warnings provided by our own worst nightmares more seriously.
These aren't operational systems, these are research systems. No need for the hyperbole. The reason for these tests is to figure out exactly these kinds of failure modes (many of which have already been predicted) and how to handle them.
"It worked in the simulation," is the embodied AI equivalent of "It worked on my machine." It's not acceptable.
How confident are you that the issues can be addressed? How confident would you need to be to endorse deploying it out in the wild? How confident would you need to be that no "unforeseen" issues would arise?
I believe it's necessary to establish these levels of confidence even before discussing specific failure cases.
We would normally discuss failures in terms of base-rate comparison, but we don't have one for "how often autonomous weapons are unaligned?" Enumerating the failure modes isn't a substitute for the empirical distribution of failure mode probabilities.
I don't think they would be talking about points if someone lost their lives.
It's pure clickbait, should be removed.
There are easy ways to clarify the article but they clearly didn't try.
"Rogue AI Destroys Normally-manned Control Tower in USAF Simulated Test"
Hell, the wikipedia on "Military Simulation" [1] is very clear that there's a very broad gradient under the section "The simulation spectrum", saying:
The term military simulation can cover a wide spectrum of activities, ranging from full-scale field-exercises,[2] to abstract computerized models that can proceed with little or no human involvement—such as the RAND Strategy Assessment Center (RSAC).
Military exercises focus on the simulation of real, full-scale military operations in controlled hostile conditions in attempts to reproduce war time decisions and activities for training purposes or to analyze the outcome of possible war time decisions.
the neutral way to say this is something like, "drone simulated killing a human operator in test", which removes ambiguity of whether the killing happened independently from the simulation, or within it.
you may not have read it that way, but plenty of people did. it's safe to assume that this was done intentionally.
the title should be changed.
Why aren't there hard limits: 'Protect our humans at all costs, protect our own assets, obey all laws of war.'? That seems like an obvious, fundamental consideration. Killing our own (and civilians) shouldn't be a matter of "points"; it shouldn't be done regardless of points.
It's possible that the speaker just didn't express it well.
(with a giant exception for the enemy)
I saw asimov speak once, and he tended towards deeper scifi than the pulp side of things.
In The Rest of the Robots, published in 1964, Isaac Asimov noted that when he began writing in 1940 he felt that "one of the stock plots of science fiction was ... robots were created and destroyed their creator. Knowledge has its dangers, yes, but is the response to be a retreat from knowledge? Or is knowledge to be used as itself a barrier to the dangers it brings?" He decided that in his stories a robot would not "turn stupidly on his creator for no purpose but to demonstrate, for one more weary time, the crime and punishment of Faust."[1][2]
“There was just enough ambiguity in the Three Laws to provide the conflicts and uncertainties required for new stories, and, to my great relief, it seemed always to be possible to think up a new angle out of the sixty-one words of the Three Laws.” (The Rest of the Robots, 1964).[3]
[2]Isaac Asimov (1964). "Introduction". The Rest of the Robots. Doubleday. ISBN 978-0-385-09041-4.
[3]https://scifi.stackexchange.com/questions/253748/how-did-isa...
lol, maybe he was like Kuttner's Gallegher and churned out pulp books that ended up being masterpieces.
feels a little different, eh?
I'd guess the "AI" was another human in a wargame, not an actual AI.