A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136
https://arxiv.org/abs/1608.04062
It's kind of an odd thing. I (random non-academic amateur) actually spent a bunch of time trying to parse the paper, which was kind of a combination of interesting ideas and incomprehensible ambiguities.
One real academic researcher also put some time into it. The good part of the paper is explained here. My guess is the problem is going from ARM to SARM.
While I'm sure most people involved think of the experience as a wash, I feel like I learned a bunch about deep learning in the process.
PS, also sad that the author did this.
The problem is not in the ARG->SARG approximation, but the bit on unsupervised pretraining. The paper could stand on its own without that section, but without that result it would have been a significantly more mediocre NIPS submission. Hope this clarifies things.
As far as the ARG-> transformation goes, maybe that's just something I don't get, I can see how one goes from sparse encoding to repeated ARG-type transformations and how this repeated application approximates the solution of a sparse encoding problem. And it is suggestive that these application look like a layers of a neural net.
But when you switch to stacking, what are you doing? Solving one sparse encoding problem then another? What analogy is there to say this works ... or that it would work better than just single sparse encoding? At that point, is it just "try it and see?"
One of the impressions I got from scanning the literature is that deep nets are kind of generally treacherous beasts - just getting a locally 1st layer may not be desirable off the bat. People have settled on backpropagation for very subtle reasons. See "Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping", Caruana, Lawrence, et. al where backpropagation finds better solutions than the "more powerful" conjugate gradient method.
Your second question of why this is a good idea is the million dollar question. Its pretty much "lets try it and see", with some heuristic reasoning thrown into the mix (its mirrors the brain, it abstracts information, etc, etc).
btw, I don't think people use early stopping anymore. It's been replaced by more powerful forms of regularization, such as dropout. The deep learning world is getting more tame, and that makes me happy.