Comment by davidtgoldblatt | Hacker News Reader

Comment by davidtgoldblatt | Hacker News Reader

Parent Full thread

davidtgoldblatt

I'm don't have a ton of expertise in the area, but I chat sometimes with people who do, so take this as a reasonably informed but lay opinion.

By and large, most uses of machine learning don't get much value from better machine learning in the sense that, given finite time and resources, you'll probably get better results on your problem by spending time on feature engineering instead of on machine learning improvements.

Some of the new deep learning techniques though have achieved good results on a large variety of problems in more of a "just plug it in" sense. My favorite example of this is this paper: http://machinelearning.wustl.edu/mlpapers/paper_files/ICML20... , which handles both sentence parsing and scene decomposition. They get competitive results on wide-ranging problem domains which have traditionally required lots of fine-tuning to do well on.

So the hope is that deep neural networks will provide improvements in the same way that SVMs did 20 years ago; that people will be able to just dump numbers into a more or less black box algorithm and get better results than they currently do. And indeed, I'm guessing people were as excited about SVMs in the 90s they are about deep neural networks today.

The advantage of deep learning is that you don't have to engineer features. It does the feature extraction for you, which saves engineers months of work.

Sorry for being unclear; this is the comparison I was trying to draw. The new techniques look like "magic" the way SVMs did when they first became popular. It triggers excitement the way it did when people realized that they could get near state of the art performance on the MNIST digits just by dumping raw pixel values into an SVM.

This is also one of its major drawbacks, though, because the features it "discovers" are often not easily interpretable. Sure, you can get good classification performance, which is sometimes good enough.

I work with a lot of scientists, though, who don't just want performance, but insight. This is where neural networks and related techniques still fall short compared to more "traditional" methods.

Avoiding the need for feature engineering is one purported advantage of deep learning. The reality is that shallow methods also work on raw data. All you need a large data set and a model whose capacity can grow with the size of the dataset (kernel methods, random features, boosting, even polynomials).

Another purported advantage of deep nets is that they require less data to train. The hope is that hierarchical representations encode knowledge more efficiently than shallow ones. This is motivated by biology and by theoretical bounds on the size of circuits for representing boolean functions. But empirical evidence on real world data does not support this hope. One can obtain just as good a predictor as any deep net by training a shallow network with a comparable number of parameters (one recent example is "Kernel Methods Match Deep Neural Networks on TIMIT").

The third purported advantage is the mid layers of deep networks can be repurposed from one application to another. There a few convincing examples of this ("Analyzing the Performance of Multilayer Neural Networks for Object Recognition" is one example). That's likely a win over shallow networks because shallow networks have no intermediate representations to speak of.

My figure of merit for any supervised learning technique is a monotonic function of

  1) test error of the final model
  2) time to evaluate on a test sample
  3) time to train
  4) human effort to train

For deep nets, every improvement in 1-2 has come with terrible deterioration in 3-4. Furthermore, shallow nets still do better than deep nets in all four respects. Which, is exactly why deep networks interest me. I'm a sucker for the underdog.

I disagree somewhat, in my opinion is rather more difficult to use. However, deep learning can solve problems that where previously insolvable. It's a step toward "real" artificial intelligence.

Reply on news.ycombinator.com