> UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation

This is a weird one. Its clearly not true (even confirmed as not true), but its is really making the rounds in all media globally right now even though it sounds like something a middleschooler made up. People are really freaking hungry for AI news huh?

The RAeS Summit notes at https://www.aerosociety.com/news/highlights-from-the-raes-fu... have a few highlights that might be of interest to the HN crowd.

As well as the part this taskandpurpose.com article is based on:

> one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation. Said Hamilton: “We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”

> He went on: “We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

It also includes some interesting stuff about autonomous vehicle security:

> autonomous cars have already been ‘hacked’ and, as the Ukraine War has demonstrated, the need for reliable supply of equipment, food and ammunition could be at threat. Why bomb a column of supply vehicles when you can simply ‘turn them off’? Or as Clarke asked: “Why shoot down a jet when you can steer the pilot’s car into a ditch before he’s even reached the airfield?”

As well as some great stuff about the military value of commercial off-the-shelf drones, and the challenges of tactical deployment of UAVs:

> “The Queen’s Dragoon Guards are fresh back from Mali,” he explained. “They took around 30 UAVs with them and we came back to two! They broke, they got lost, they deteriorated with the conditions and couldn’t be fixed - there aren’t many trees in Mali but they found them!

The media is dying to build up the "AI is going to eradicate humanity" narrative, such as in this *very* misleading headline (nobody was "killed", but it sure sounded like someone was), which ultimately is going to be another big giant mess of garbage information your aunt believes, all while OpenAI laughs all the way to the bank. I think reasonable voices who aren't set to make billions from the burgeoning "saving us from the AI apocalypse" industry need to speak up more.

Except that it never happened.