Hacker News Reader

Comment by maxander | Hacker News Reader

So someday, Hypothetical Nation#1 captures one of Hypothetical Nation#2’s optically-guided missiles that uses a neural network to distinguish friend from foe. N#1 technicians download the network weights and use this to generate perturbatory paintjobs for their fighter jets, making N#2’s missiles recognize N#1’s planes as various models of N#2’s civilian aircraft. Before N#2 can refit their missiles with a retrained neural network, N#1 launches a massive offensive and decisively takes control of the Hypothetical South China Sea, or something.

Do I have that right?

View on HN

In theory that's possible, but every ML researcher is aware of adversarial examples. In any high-stakes use case with a high chance of adversaries, I doubt people will use neural networks if classical algorithms will suffice, and if they do, they will definitely test and design for the existence of adversarial examples.

Or Uber technicians will download weights from a Google car, and derive an adversarial paintjob for billboards or other Uber cars.

It's more likely something that would make civilian smart tech useless against the top 5-10 militaries.

A military guidance system would overtrain a network using a supercomputer and then cryptographically make it stupider so they could give every missile 5 individually generated networks that vote, forcing attackers to compromise 3 networks per missile. (There may be some back and forth for a while, but here you can create a diversity of networks that makes the ensemble hard to attack.)

So they'll just shoot them down with lasers cause that's less energy intensive than running the compute node to crack missiles in real time.

You just need a second neural net that classifies adversarial and non-adversarial objects.

And of course a third network to classify the adversarial objects that slip through the second.

Reminds me of the record player chapter from GEB [1]

[1] https://genius.com/Douglas-hofstadter-contracrostipunctus-an...

After a couple of iterations, an adversarial object will become the adversarial object for humans too. That is a military plane disguised as a civil plane. It is probably forbidden by some convention.

Reply on news.ycombinator.com