Adversarial.io – Fighting mass image recognition

My feeling is that even inception trained on imagenet with a different random initialization would probably not fail (or at least be much less likely to fail) when exposed to these adversarial examples.

If these are hacking the specific, underspecified, realization of the classifier (I.e. the set of public weights trained from a specific seed and visit order through the data), the adversarial examples are probably just as fragile as the classifier.

Not sure on published research, but anecdotally I've played with this and found it depends on the technique used. The most simple attacks tend to be model specific. This often means that it won't work at all on another model or will work but less effectively (the confidence of the adversarial target will increase and the true class will decrease but not necessarily to a degree that will change the classification). It can also depend on the dataset in addition to the model architecture.

The more simple ones don't even really work after basic transformations (like rotating, scaling, etc) on the target model, so those attacks are often brittle. But there are lots of techniques and some of them are more robust across more models and transformations. This sometimes has a tradeoff of causing the manipulation to the image to be more noticeable to the human eye.

Adversarial attacks are a bit of a cat-and-mouse game between new attacks and new attempts to find where they fail.

Transfer of adversarial perturbations between models is one of the main avenues of "black box" (where you don't know the target model's weights) attacks. Perturbations don't translate between models 100% of the time or anything, but many attacks are surprisingly transferable. There are also methods to make perturbations more transferable, for example by finding an attack that is effective against an ensemble of models, you increase the chances that it will transfer to an unseen model.

https://arxiv.org/pdf/1611.02770.pdf

If it's opensource, there's no need to wait on the business interests of a trillion dollar company to align with your wishes. Camera app developers can be made aware of it and add it to their apps. If there are app developers on HN, they can create pull requests to their favorite apps and add the feature.

That's the power of opensource.

Definitely agree that there will be cases in which a computer vision operator ships features specifically intended to create a new automatic defense.

One thing that seems unique to technologies that are mostly just statistical learning is that each new manipulation approach can basically widen the distribution of possible inputs. In particular, I'm thinking that as more obfuscation and protest technologies are made public like this, the distribution of "images of faces available for computer vision training" becomes more complex. That is to say, whenever a adversarial tool creates a combination of pixels that's never been see before, if that "new image" can't be reduced back to a familiar image via de-noising or pre-processing, the overall difficulty of computer vision tasks increases.

All a long winded way of saying, I think for ML systems, there's a unique opportunity to "stretch the distribution of inputs" that may not exist for other security arms races.

Totally agree that economics of the arms race(s) will a huge factor in determining how much an impact obfuscation and protest can have.

I don't really think this is true for two reasons. First, the app doesn't add anything that someone training a classifier couldn't just do themselves. If I want to train my network against adversarial inputs, I can just generate them myself at training time on my own training data - it's not particularly helpful for me to have a bunch adversarially-perturbed but unlabeled images. Second, and more importantly, it's not sufficient to just train on adversarial examples taken from another network. You might learn to be robust against the specific weaknesses of that other network, but your new network will have its own idiosyncratic weaknesses. To be effective, adversarial training (https://arxiv.org/pdf/1706.06083.pdf) needs an adversary that adapts to the network as it trains. In other words, you need your adversary to approximate your current weaknesses at each step of training.

Right. I’m naive in assigning more capability to these models than they posess.

I found a quote from Geoff Hinton where he talked about this last year.

From [1]: “I can take an image and a tiny bit of noise and CNNs will recognize it as something completely different and I can hardly see that it’s changed. That seems really bizarre and I take that as evidence that CNNs are actually using very different information from us to recognize images,” Hinton said in his keynote speech at the AAAI Conference.

“It’s not that it’s wrong, they’re just doing it in a very different way, and their very different way has some differences in how it generalizes,” Hinton says.

[1] https://bdtechtalks.com/2020/03/02/geoffrey-hinton-convnets-...

> Because there is no I in AI.

This. In a nutshell every sort of algorithm we call "AI" today is reductive pattern matcher. This limitation isn't due to computational capacity or even, IMO, algorithm design, but due to our collective lack of understanding of how intelligence itself works. We'll get there eventually, but not for a long while.

> Because there is no I in AI.

This a common refrain but fairly obviously untrue. It assumes there's some secret sauce in human brains that makes us "intelligent" whereas AI is "just a machine".

It's pretty clear that human brains are just programs. Extraordinarily complicated highly optimised programs, sure. But nobody has even found a shred of evidence that there's anything fundamentally different to programs in them.

Thinking otherwise is along the same lines as thinking that animals don't have feelings.

Every time there's an advance in AI the "it's not really intelligent" goalpost shifts. Clearly intelligence is a continuum.

> At a mile high conceptual level, AI is nothing but a program created by a computer based on the data it is provided

Your brain is but a preprogrammed, biological computer that reacts to data obtained from its interfaces and attach peripherals.

We'll end up in a similar cat&mouse game as online "pirates" been at for a long time. Developers create something to break the AI, the AI adopt because they figure out the noise profile, developers change the noise profile and the AI has to adopt again.

The poster is not wrong. This attack only works against a specific model trained against a specific dataset. It does not work against models that understand that a animal face is made up of sub features like a nose, two eyes and a mouth.

In the end, this is a completely useless exercise and will not have any impact on mass image recognition. For this to even work the attack needs to be tailored to the exact weights in the neural network that is being attacked.