Does Deep Learning Have Deep Flaws?
kdnuggets.com
kdnuggets.com
This isn't to take away from the research; the most interesting result was just how close to valid inputs these erroneously classified images are.
But again, this isn't some fatal flaw. This summary completely neglects the fact that the paper also recommends that -- just like distorted images are added to training sets today (you wouldn't want something common like optical aberration from the camera lens screwing up your classifier) -- in the future, these adversarial examples should be added to training sets to mitigate their effects.
> In some sense, what we describe is a way to traverse the manifold represented by the network in an efficient way (by optimization) and finding adversarial examples in the input space. The adversarial examples represent low-probability (high-dimensional) “pockets” in the manifold, which are hard to efficiently find by simply randomly sampling the input around a given example. Already, a variety of recent state of the art computer vision models employ input deformations during training for increasing the robustness and convergence speed of the models [9, 13]. These deformations are, however, statistically inefficient, for a given example: they are highly correlated and are drawn from the same distribution throughout the entire training of the model. We propose a scheme to make this process adaptive in a way that exploits the model and its deficiencies in modeling the local space around the training data.[1]
I've seen several articles citing this paper as proof that deep learning is deeply flawed, yet they all seem to miss the point you make above.
The other interesting result is that the neurons are not in fact individual features you can just grab and drop into another algo--the entire space defined by the model works together, through all the layers. Honestly, that was a more interesting result for me, although I don't know that it negates anything, I've just got to stop telling people that the individual units are features.
Not that deep learning is the end-all-be-all of machine learning--it's not. It's just that this paper isn't saying what reporters are saying it's saying... As per usual?
Never let the truth get in the way of a good story...
What this result says to me is that there are really useful features of the data that can successfully classify images that humans are totally unaware of! And that's neat.
[0] http://cs.nyu.edu/~zaremba/docs/understanding.pdf [1] http://puu.sh/9B4eG/1e9f7eb56b.png
> Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous [...] Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation [...] the same perturbation can cause a different network that was trained on a different subset of the dataset, to misclassify the same input.
It's an interesting outcome -- but there are many deep-learning approaches and many different benchmarks, so it will be important to see if this is a misleading anecdote or indicative of a systematic problem.
Can I turn all digital pictures of me into 'adversarial examples', so the eye of sauron can't identify me from pictures?
I'm sure it's not as simple as that, presumably any algorithmic modification to an 'adversarial' nature can be countered by other algorithms.
But I predict a new realm of 'arms race' here in the future.
Like if you fit 5 close-to-linear 2d points with a 4th order polynomial, you'll overfit. Change the data slightly and you'll still overfit, but your fit will be very different.
http://slashdot.org/story/14/05/27/1326219/the-flaw-lurking-...
The comment I recalled was written by someone with the handle "presidenteloco".
It is also rather striking that these DLNs seem to be tricked by what we would typically think of as noise.
The statistical naivete of the field led to all sorts of inadvertent mixing of training and test set data which generated a lot of spurious claims for solving the problem. That is until one attempted to find those decoys and they were always found. This led to the creation of the CASP competition to weed this out and the field finally moved forward.
http://en.wikipedia.org/wiki/CASP
The key similarity to what I described above is that adversarial search is done posterior to the training of the deep neural network. That makes all the difference in the world IMO. These adversaries may just be strange, otherwise hard to reach bad neighborhoods in image space without using a roadmap. Or they may be an unvaoidable consequence of the curse of dimensionality.
http://en.wikipedia.org/wiki/Curse_of_dimensionality
But given that neural networks have a gradient, it doesn't shock me that it can serve as a roadmap to locate a set of correlated but seemingly minor changes to an example in order to flip its classification. Doing so is simply back-propagation with constant weight values to propagate the gradient to the input data itself - literally a couple lines of code.
IMO there are two interesting experiments to do next (not that anyone will take this seriously I expect, but ya know, hear me now, believe me later):
1. Characterize the statistical nature of the changes in input images and then use those summary statistics as the basis of an image altering algorithm to see if that can be used to flip the classification of any image on its own. If it can, be afraid, your driverless car may have blind spots. If not, then this is probably just a narrower form of overfitting.
2. If it's likely overfitting, attempt an expectation maximization-like fix to the problem. Train the network. Generate adversaries, Add them to the training set, train again and then lather rinse repeat until either the network can't be trained or the problem goes away.
Expensive? Yes. But you're Google/Facebook/Microsoft and you have lots of GPUs. No excuses...
Failing that, the above is on my todo list so I'm throwing it out there to see if anyone can poke holes in the approach.
it was an special shock when I learned about ensemble methods (I think they were just called "combined servers" at the time) at CASP and saw that all our hard work (manual alignments, lots of expert analysis of models, etc) wasn't really better (far worse in fact) than a few simply trained ensemble systems that memorized what they were bad at and classified their predictions with the appropriate probabilities.
See also: http://www.nature.com/nchem/journal/v6/n1/nchem.1821/metrics... http://googleresearch.blogspot.com/2012/12/millions-of-core-... (note, 4 of the 6 projects awarded specifically involved physical modelling of proteins and the fifth was a drug-protein binding job) http://research.google.com/archive/large_deep_networks_nips2...
none of the above are coincidental: the first two links are specifically because I went to Google to use those GPUs and CPUs for protein folding and design and drug discovery. The third project is now something I am experimenting with.
Here's an example of what drove my work back then:
Look at the energies and RMSDs (a measure of distance from the native structure) of melittin in these two papers:
Table 2 in http://onlinelibrary.wiley.com/doi/10.1002/pro.5560020508/pd...
and
Table 1 in http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1260499/pdf/biop...
In the first paper, the energy is higher, but the RMSD is lower. In the second paper, the RMSD is higher, but the energy is lower. How did this happen?
Well, in the first paper, phi/psi angles are set directly from a library of sequentially homologous dipeptides to pentapeptides that INCLUDES MELITTIN. So, by the time you get to tripeptides, you're nearly guaranteed to just be outputting the native conformation phi/psi angles over and over again. And this paper is just one of many to make basic mistakes like this.
As young turk back then, I got into a rather long and vigorous online argument with one of the founders of CASP who insisted the first paper was a partial solution to the protein folding problem. And I suspect that argument influenced the subsequent creation of CASP.
Anyway, it's been nice rehashing my post-doc glory days(tm), but we no longer have any excuses here. We have the tools, we have the technology...
As I quoted in my other comment, the paper suggests doing exactly that.
"The above observations suggest that adversarial examples are somewhat universal and not just the results of overfitting to a particular model or to the specific selection of the training set. They also suggest that back-feeding adversarial examples to training might improve generalization of the resulting models."
20 years ago I did this for linear models for protein energetics (also known as knowledge-based potentials or force fields), adding the decoys then refitting the parameters ad nauseum. What I eventually arrived at was the invalidation of every single energy model and force field in use for protein energetics (yes I really reverse engineered just about everyone from Michael Levitt to George Rose to AMBER, CHARMM, and ECEPP). This was an unpublishable result according to my post-doc adviser at the time so it never got written up.
In retrospect, he was utterly wrong. So I am really curious what would happen here if this were attempted with these much more complex models.
That said, I'm a bit surprised you found what you did about AMBER (and other force fields), or rather, that you didn't publish. The cornell et al force field was later acknowledged to have serious problems with protein folding, but a number of improvements have been made since then.
Anyway, I would have happily published that result with you (I worked with Kollman, have worked with Baker and Pande, and desperately want to see the force fields improve using machine learning). There was a guy at BMS who was working on this back in the day ('99-2000) who was using ML and the AMBER folks trashed him because they believed the force field's transferrability from small molecules to proteins was valid (in many ways it was, but it got some key details wrong).
Ifd you think there is a straightforward machine learning for force field problem that can dramatically improve ab initio folding with distbelief and exacycle, let me know. It shouldn't be hard to figure out my email address if you look at the papers I cited and do some basic set operations :-)
Also, the really interesting thing was that adversaries generated for one network topology/data set were still adversarial even for other network topologys/data sets, which might imply that the nature of the adversaries is universal rather than highly specific to that exact network trained
Any algorithms have weaknesses. It's a matter of being aware of them and compensating for them in your model, possibly by using another model.
The problem is that the adversarial examples they are able to come up with are very close to the original images, so this means the smoothness assumption seem to be invalid for deep learning models. As they put it in the paper : "Our main result is that for deep neural networks, the smoothness assumption that underlies many kernel methods does not hold."
It's going to be interesting to see what happen when other researchers try to replicate the results for other models and datasets.
To accomplish that, for example, they could take perturbations of the input and then e.g. take a majority majority vote.
However, maybe there's a middle ground. I.e. maybe we don't need the more esoteric features that SIFT uses, but it just makes sense to do edge detection, and say a Fourier transform for audio.
Normalization and other data transforms are still required for discovery of features.
The impl is a convolutional RBM. That being said, where's your deep learning library? ;)
The documentation is still being worked on. If that's the only thing you can cherry pick from a new project, I must not be doing too bad.
Edit: Clarified. I know you're just a troll account, but I'll throw this out there anyways, I think anyone who judges the quality of a lib based on a 2 second reading isn't qualified to judge much. A lot of it does rely on me with the documentation, but I'd love to chat with you one on one if you think I'm not qualified. Trash me all day, I'll either learn something or embarrass you. Both are fine with me ;).
I would probably recommend Torch at this point. The incentives don't exist for the experts to make really good open source projects and spend all the time required maintaining them and helping people with them.
You're right about this which is why I started a company around it.
I've already talked with andrew ng and yoshua bengio. My incentives are different from there's, however, I do have their blessings to continue doing this.
I walked in to this expecting skeptics. That being said, I love deep learning as a field and will be implementing every possible neural net I can. Since my incentives are different, I can explore the different use cases with customers and help further the field in different directions that might not make sense for say baidu, facebook ,or google.
More specifically, adding SIFT or edge detection to your raw pixel input will almost always strictly improve a deep model's performance (though they might be redundant) at a not-particularly-large computational cost.
It wouldn't solve the adversarial example problem though, except to the extent that it makes calculating gradients harder.
Yes and just to clarify - this seems like an indication that however useful, deep learning can't follow through with that promise. And I think the ability to do this the key thing - all the approaches work at some level but without this "automagicity", each become hostage to brow-wrinkling experts who become the only one to understand the black-magic of algorithm tuning.
Neutral networks are not perfect solutions, they are solutions that get an organism to reproduce successfully.
Read any book on color vision humans have similar problems, yet for the most part we see things, and realize that clouds are just clouds and not faces, except for the religious, they lose their shit when faces appear in clouds.
http://en.wikipedia.org/wiki/Lost_in_the_mall_technique
http://en.wikipedia.org/wiki/Memory_conformity
Not sure what you mean by beliefs.
There was also an experiment on split-brain patients (the connection between the left and right hemispheres is severed) where they'd show a command like "WALK" to the patient's right hemisphere only. They'd get up and walk. But since language is often localized to the left hemisphere, if you talk to them you are talking to the left hemisphere only, which did not see the command. Instead of saying something like "I don't know", they would make up a plausible reason to get up like "I'm getting a drink".
Does that count? It exploits what we know about the visual system (half of the visual field goes to each hemisphere) and localization of a particular function (language) in a patient with a specific disability (their corpus collosum is severed, which is mostly asymptomatic) to produce a completely strange result (invention of a motive).
In audio, we have the Shepard tone (http://en.wikipedia.org/wiki/Shepard_tone) as an auditory illusion.
Not sure if you consider that "corruption".
http://psych.hanover.edu/JavaTest/CLE/Cognition/Cognition/fa...