Adversarial Examples Are Not Bugs, They Are Features
arxiv.org
arxiv.org
It's intriguing how much focus there is on adversarial examples. You don't need adversarial examples to make a deep network fail - in a sense that's overkill. Just point the poor deep network at a sequence of images from the real world -- images from a self driving car, security camera, or webcam. You'll see it make spontaneous errors. No matter how much training data you gave it.
The field will advance when/if practitioners recognize that classifying pixel patterns in isolation isn't sufficient for robust visual perception, and adopt alternative neural network designs that can interpret what they perceive in light of (no pun intended) context and physical expectations.
It worked for our prototype.[0]
Why? Ambient context.
Why? Ambient context.
That was a really great way to get the point across, Especially because I still sometimes think it's real, even when I know the context.
As an aside - from the paper it looks like you worked at Brain Corp a few years back. Any thoughts on them and what they're doing these days? I'll be looking for a job again soon and i see a lot of ads for them.
This seems to be only a very small step forward from Minsky's negative result about "perceptrons".
They're not even close to structural or training algorithm complexity of natural neutral networks yet.
DNN is essentially one of these with more layers than just typical 4 for MLP, because we figured out a way to propagate error and training gradients. (Plus a few important but interesting details.) They are not really qualitatively different according to math they use... The main difference is use of gated or not differentiable activation functions with various ways to compute approximate gradients when faced with this feature. Especially convolutional nets are similar to MLP.
We are essentially building a frog with better and better visual perception in the hope that it could become a taxi driver. It will become a totally amazing super-frog with super-vision, but it's still just a frog with frog-like visual perception and limits. Using pre-attentive feature recognition stage equivalent for complex object recognition can fake human like object recognition when we force it, but it's wrong approach. We get these catastrophic failures because we hit the limits.
Features seem to exist independently from one another in the early processing stages of human perception. They are not associated with a specific object either. Human perception is not gradually turning features into objects like we do in deep learning. Properly distinguishing feature integration from detection and how to do it is a open question.
> Learning multilayer convolutional representations of statistical features is roughly equal to taking few first few layers in visual cortex and stacking them.
No, it isn't roughly equal the first few layers of visual cortex. The first few layers of visual cortex have substantial feedback connectivity from higher areas which affects the responses of even the most peripheral parts. (Citations in our arxiv preprint linked above.). Most of the brain has more feedback connectivity from elsewhere than feedforward ascending connectivity. This qualitatively affects activations.
>We are essentially building a frog...
I suspect frog vision is far more robust than anything we are "essentially building".
> Features seem to exist independently...
Please have a close look at some modern visual neuroscience. Or speak to an good honest electrophysiologist.
> The field will advance when/if practitioners recognize that classifying pixel patterns in isolation isn't sufficient for robust visual perception
But this, well, it is very clearly sufficient, and we have well accepted results showing this. It just won't work on practice. That probably means the change will be full of fighting while the old ways still work, and lots of failures and unexpected successes.
Recent success of gpt-2 indicates otherwise.
So thank you.
Can someone explain to me the difference between this statement and the notion that AI performance in general is completely subjective?
We're trying to train models to do certain things. It doesn't matter if you call them human-centric or not. The important thing is that we have a goal for training. Adversarial examples force models to do other things, i.e. behave in a way that defies the original goal. How is saying "no, misclassifying those bad examples is okay" different from moving the goalposts?
Their process of retraining doesn't make any sense to me either. So what if the model trained on mislabeled data has some degree of accuracy on real data? This just shows that there is some internal symmetry involved.
The conclusion states:
> Overall, attaining models that are robust and interpretable will require explicitly > encoding human priors into the training process.
I feel that is true, though another part of the solution IMO lies in coming up with classifiers that can do more than output a probability alone. I agree that classifiers being sensitive to well-crafted adversarial attacks is something that can't be avoided (and perhaps even shouldn't be avoided at the train-data level), but the problem lies mainly at the output end. As a user, the model gives no insights towards how "sure" it feels about its prediction or whether the inputs deviate from the train set (especially in the useful non-robust feature set). This is especially a problem given that we stick softmax on almost all neural networks, which has a tendency to over-estimate the probability of the rank 1 prediction which confuses humans. Most adversarial attacks show [car: 99%, ship: 0.01%, ...] for the original image and [ship: 99%, car: 0.01%, ...] for the perturbed image.
Using interpretability and explanatory tools to inspect models is a good start, though I'd like to see more attention being given to:
- Feedback with regards to whether a given instance deviates from the training set, and to which extent
- Bayesian constructs w.r.t. uncertainty being incorporated, instead of only probabilities. Work exists that tries to do this already [1,2] with very nice results, though is not really "mainstream"
[1]: https://alexgkendall.com/computer_vision/bayesian_deep_learn...
[2]: https://eng.uber.com/neural-networks-uncertainty-estimation/
We just do not know how high level structure of a mind looks, best we have is some sort of data compression entropy model. That's obviously not enough.
Adversarial training model is probably closer (e.g. A3C) but it's not detailed enough either.
Value and policy loss are extremely blunt tools to evaluate an actor or critic, for example
Would be interested to know if your experience differs and in which industries.
Robustness is a systems statistical notion of amount or degrees of freedom of state perturbation required to change output, also taking into consideration the magnitude of change. It is related to but not same as system theoretical stability. There's nothing human about the definition. Robust features need not be human derived.
The desired degree of robustness vs absolute accuracy or precision or bias trade-off is human specified but generally the trade-off is not huge between these variables.