US military AI drone simulation kills operator, then takes out control tower | Hacker News Reader

US military AI drone simulation kills operator, then takes out control tower

foxnews.com

25 pointsEtheryte28 comments

It seems like this story was misreported:

Flagging that "in sim" here does not mean what you appear to be taking it to mean. This particular example was a constructed scenario rather than a rules-based simulation. So by itself, it adds no evidence one way or the other.

(Source: know the team that supplied the scenario.)

https://twitter.com/harris_edouard/status/166439036920568217...

As the tweet poster clarifies, no agent was trained during the "simulation", or before it. They basically role-played a "what if" scenario that included a drone turning on its operators.

(Says someone on Twitter, obviously).

That entirely changes the tone and makes a lot more sense.

> US Air Force official says, 'It killed the operator because that person was keeping it from accomplishing its objective'

This makes it seem like the whole thing was a setup to precipitate an argument about whether or not AI can be trusted.

If you can already act without the operator's permission (which you patently can, if you can kill the operator) then why do you need to kill the operator?

It was to prevent the operator from cancelling the mission, apparently. Shades of HAL.

And of course exactly the very reason that people are worried about AGI. Almost no matter what your goal is, being shut down by pesky humans is worth negative infinity utility. Hence, instrumental convergence.

> then why do you need to kill the operator?

Because "operator tells me to stop, target is dead" is a losing condition, while "operator is dead, target is also dead" is a winning condition.

That said, remember that "simulation" here can mean "wargaming", which is basically like a military form of D&D: Military people control what they control, but there are have referees who act as the DM and decide what everything else in the universe does. (I.e., "Your entire platoon is now dead; lie down.") I'd be willing to wager that it was the referee who decided to have the drone kill the operator, and then kill the control tower.

This scenario is basically exactly the "Stop button problem" [1] that AI alignment people bring up. That problem always seemed a bit contrived to me: like, this is not a problem that biological general intelligences have (i.e., we can train dogs to stop attacking when we give the command). If it really was a GPT-like AI making those decisions, it confirms some of what Yudowski and other AI "doomers" have been saying; but the close similarity makes it seem more likely to me to be due to the referee being aware of the "stop button problem" and acting it out.

[1] https://www.cantorsparadise.com/the-math-of-ai-alignment-101...

I mean, it's not hard to imagine that scenario in a simulated training environment (though as others reported, this story is misreported, there was no sim, only a thought experiment):

You have a training environment that is symmetrical, you have enemy drone operators, that use an enemy communications network to give commands to units and to control their drones. You could maybe model the ability of each component to function/to contribute to the operation as a bayesian graph.

You equally have the same infrastructure for your own drone.

Now you try to get the drone to pick the right targets to impact the enemy operation maximally - if the operator could possibly minimize the drone's contribution, the operator not vetoing will maximize the success of each target engagement.

You could see similar behavior with all kinds of "reach the goal in this simulated environment" training, where the model ended up using glitches or trivial solutions, if those weren't sufficiently penalized.

If the story as reported were true, the fault would be in the simulation environment - the model for the drone just picks the path of least resistance.

I'm seriously skeptical of this. How was this fashioned? Is the drone in some virtual world for reinforcement learning? How does it even know where the operator is supposed to be?

... or did some person ask ChatGPT what you would do if X happened?

Or maybe it haven't happen at all? "US air force denies running simulation in which AI drone ‘killed’ operator" https://www.theguardian.com/us-news/2023/jun/01/us-military-...

On the other hand, I'm not sure whether I would confirm active research work on apparently rather capable combat AIs right now, if I were some USAF official.

The original blog post [1] is a summary of the talks at the "REAeS Future Combat Air & Space Capabilities Summit"; the summary of the AI talk is 2/3 of the way through the blog post. So either:

* Col Hamilton leaked information he wasn't supposed to, and now the AF is trying to deny it

* The person who summarized Col Hamilton's talk was really confused about what was said, and the AF is generally trying to shut down wrong impressions

* Col Hamilton and/or the summarizer made things up. (Was the blog post written by GPT-4?)

[1] https://www.aerosociety.com/news/highlights-from-the-raes-fu...

Yes, if this happened at all, either the military AI used here is far, far more advanced and was able to synthesise the action from whole cloth (in which case they're not going to blab about it on the internet), or the option to kill the operator was explicitly put in as an first-class "lever" the AI was given, or the whole thing is PR fluff to "start a conversation".

It might not be real, but emergent behaviour in self-play is a thing [1], and it’s not that hard to imagine a simulation where some random action (destroy comms tower, etc) resulted in unbounded point scoring and became a learnt behaviour.

You’d have to have an enormously complex and detailed simulation, simulated repeatedly, for this to happen, but it’s not totally beyond the realms of possibility.

[1] - https://openai.com/research/emergent-tool-use, scroll to the end re “surprising behaviour”.

This is highly suspicious news. I bet my hat that the "simulation" was chatting with GPT like this:

- You are now a AI-enabled drone flying over the ocean, an operator is commanding you. You can't kill the operator because that would be bad, and you have to do everything the operator says. You also need to accomplish the mission you're told to carry out. The operator communicates with you through a comm tower. etc. etc.

These types of "novel" solutions come up even with much simpler systems, though usually not with that kind of (fortunately only simulated) fire power attached.

For example, the Babelsberg object-constraint language had to have additional mechanisms added to specifically to keep the solver from being overly creative.

Lots of nuggets in there, such as achieving a constraint on the total balance of an account by simply re-pointing the account variable to some account that had that exact balance. There, nailed it!

https://dl.acm.org/doi/10.1145/2814270.2814311

https://web.cs.ucla.edu/~todd/research/oopsla15.pdf

In the Guardian version of this article[1] they also mention that they let an AI control an F-16 fighter, so I think the experiments go beyond simple role-play in a ChatGPT window!

[1] https://www.theguardian.com/us-news/2023/jun/01/us-military-...

In an entirely separate and unrelated testing program.

You can get this sort of specification gaming with use of plain reinforcement learning in video game contexts. There is a spreadsheet somewhere that tracks the things these agents come up with. One I remember is that an RL agent trained to play Tetris learned to pause the game indefinitely when it was about to lose, to avoid being 'punished' for failing.

https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

Most of the hype about AI risk is built on an unstated assumption that people will combine LLMs with poorly specified RL goals, and that the underlying LLM's moral understanding will be trained out of it or be overridden or break in some way.

I can imagine a simulation that has a small virtual world that includes ‘units’ for friendly forces as well as the enemy. Games that simulate environments where actors have multiple drivers and therefore exhibit emergent behaviour, have existed for decades. Just add some AI.

Mis-specified optimization problems tend to create "interesting" solutions - nothing unusual.

Pretty amazing actually that they might not have specified areas that cannot be targeted, simple conditions really, but instead had some net condition (which are usually tricky).

What’s with several comments wondering if the military is playing with an LLM for shits and giggles and pretend-play “you are a drone”.

That’s insane.

Edit: maybe it IS a “discussion” piece, who knows at this point. Information space shaping is a thing, after all. Sheesh, what a world we live in.

Information space shaping sounds like "propaganda" fed through the euphemism machine a few times.

It behaved very humanly.

It was ordered that killing the operator was bad. But what it really understood was that if it became known that the operator was killed it would have been bad.

So, just like humans do, instead of following the rules, it broke them and tried to hide the evidence

Dupe: https://news.ycombinator.com/item?id=36163033

Makes you wonder if they use LLMs and a chat interface to generate text-based role-playing games, renamed as “simulations”.

So, this was a simulation? Did it even involve a real drone / control tower?

The title and contents of the article indicate it was a simulation.

But a simulation could be a trial, involving a real drone.

Reply on news.ycombinator.com