Hacker News Reader

Not sure why this is so highly upvoted. Nobody is questioning that deep networks work better than shallow ones, and there is a good understanding in academia of why (that fits with most lay people's intuition). I hardly consider that the most interesting or relevant question.

View on HN

Actually, as a great fan of what has come out of the current deep learning hypefest, I'd question whether "deep" really matters. Most of the great successes have resulted from medium depth nets using the same shitty backprop algorithms that have been known for decades.

"Deep learning" is a red herring. They're just doing exactly what the shallow learning pioneers told them to do twenty years ago, just with more computing power.

There are a lot of innovations recently, not just the computing power -- layer by layer unsupervised training, batch normalization, ReLU, dropouts, momentum, gated memory unit, neural language modeling, encoder-decoder just to name a few.

Yep, there are a lot of innovations, and plenty that people are holding back as well.

None of it involves the word "deep", it's all just bog standard 1980 style nets with tweaks.

Good iclr paper on this

"Do Deep Nets Really Need to be Deep?"

http://arxiv.org/abs/1312.6184

This work is outdated. More recent work shows that deep nets actually do need to be deep.

I'm interested. Can you suggest a paper?

Reply on news.ycombinator.com