Hacker News Reader

Comment by mbq | Hacker News Reader

Since the fall of AI, there are two groups of people in this topic -- one trying to make some reproducible, robust results with well defined algorithms and second importing random ideas from the first group onto some questionably defined ANN model and getting all the hype because of the "neural" buzzword. "Deep learning" is actually called boosting and has been around for years.

View on HN

Unsupervised pre-training is fundamentally different than boosting.

Boosting is a clever way of modelling a conditional distribution. The insight behind the success of pre-training is that, for many perceptual tasks, having a good model of the input (rather than the input->output mapping) is key.

I have no delusion that the algorithms that work for training deep networks are anything like what the brain actually does, but I don't care. There are many tasks where deep neural nets are state of the art.

Not to argue with you, robrenaud, but Hinton himself writes in their 2006 paper 'A Fast Learning Algorithm for Deep Belief Nets':

The greedy algorithm bears some resemblance to boosting in its repeated use of the same “weak” learner, but instead of reweighting each data vector to ensure that the next step learns something new, it re- represents it.

I guess that most people however would not think of this interpretation of greedily pretraining deep networks :). (I wonder if mbq had this in mind).

In the same article your point about good models of the input is mentioned, too (only copy&paste a small part of the paragraph):

Unsupervised methods, however, can use very large unlabeled data sets, and each case may be very high-dimensional, thus providing many bits of constraint on a generative model.

The 2006 paper is really an amazing read in my opinion.

Boosting selects successive weak learners for the same classification problem, but under a changing distribution/weighting of the input space. Deep learning stacks complex models to create increasingly abstract representations. All I can really imagine them having in common is (1) they're both families of machine learning techniques and (2) they both (roughly) involve a collection of models, albeit in very different ways.

You're talking about Adaboost; general boosting can use any models and the only idea there is that it adds new model to fix residuals of the current chain. BTW "increasingly abstract representation" is a perfect example of this meaningless PR ANNs are built of.

No.

Deep learning is not about fixing the residuals of the current chain. Deep learning isn't even about residuals in the first place. It's about (1) finding good representations of your data (aka feature learning), (2) then adding a discriminative model on top and then (3) tuning everything. There is no relation to boosting at all.

Deep learning is not boosting at all. Deep learning is about composing trainable modules. Adding a layer f(x) to a layer g(x) to get h(x) = f(g(x)). Boosting creates a final classifier that is a weighted sum of the base classifiers, or something like h(x) = a * f(x) + b * g(x). Composition is what Professor Hinton means when he says "re-represent the input" and other similar phrases.

Reply on news.ycombinator.com