Avoiding the need for feature engineering is one purported advantage of deep learning. The reality is that shallow methods also work on raw data. All you need a large data set and a model whose capacity can grow with the size of the dataset (kernel methods, random features, boosting, even polynomials).

Another purported advantage of deep nets is that they require less data to train. The hope is that hierarchical representations encode knowledge more efficiently than shallow ones. This is motivated by biology and by theoretical bounds on the size of circuits for representing boolean functions. But empirical evidence on real world data does not support this hope. One can obtain just as good a predictor as any deep net by training a shallow network with a comparable number of parameters (one recent example is "Kernel Methods Match Deep Neural Networks on TIMIT").

The third purported advantage is the mid layers of deep networks can be repurposed from one application to another. There a few convincing examples of this ("Analyzing the Performance of Multilayer Neural Networks for Object Recognition" is one example). That's likely a win over shallow networks because shallow networks have no intermediate representations to speak of.

My figure of merit for any supervised learning technique is a monotonic function of

  1) test error of the final model
  2) time to evaluate on a test sample
  3) time to train
  4) human effort to train

For deep nets, every improvement in 1-2 has come with terrible deterioration in 3-4. Furthermore, shallow nets still do better than deep nets in all four respects. Which, is exactly why deep networks interest me. I'm a sucker for the underdog.

This is also one of its major drawbacks, though, because the features it "discovers" are often not easily interpretable. Sure, you can get good classification performance, which is sometimes good enough.

I work with a lot of scientists, though, who don't just want performance, but insight. This is where neural networks and related techniques still fall short compared to more "traditional" methods.

Sorry for being unclear; this is the comparison I was trying to draw. The new techniques look like "magic" the way SVMs did when they first became popular. It triggers excitement the way it did when people realized that they could get near state of the art performance on the MNIST digits just by dumping raw pixel values into an SVM.