Just one more to add... dropout!

That was a big deal, and I think pretty much eliminated greedy layerwise pretraining in the "we have plenty of data, but can't generalize well" case. Good initialization rules help too, but are mostly heuristic and problem dependent to my knowledge.

For interested parties, I will again plug my slides: https://speakerdeck.com/kastnerkyle/euroscipy2014

The last few slides have a kind of "survey list" to get up to speed with modern deep learning approaches for images. I also put the slides on github at http://github.com/kastnerkyle/EuroScipy2014 , which hopefully preserves the hyperlinks where speakerdeck does not.