Hacker News Reader

Comment by ignifero | Hacker News Reader

Might this be due to the explosion on neuroscience publications that are not related to artificial neural networks? There haven't been major breakthroughs in ANNs lately

View on HN

in machine learning, the 'big' thing about neural networks at the moment is deep learning via some sort of unsupervised pre-training. there's be a lot of work published on this recently.

http://deeplearningworkshopnips2010.wordpress.com/

Hmm, could you give us a TL;DR on how this differs from model selection in statistics, e..g, choosing the structure of an HMM?

A good intro page with lots of links: http://deeplearning.net/tutorial/

A lot of the newest work is under a few different names: "deep learning", "convolutional deep networks", "unsupervised feature learning", etc.

Here's a great talk by Andrew Ng of Stanford, who's a recent convert to this area: http://www.youtube.com/watch?v=ZmNOAtZIgIk

(Note that this is quite a one-sided view of things, but it does convey the excitement of deep-learning researchers and the potential of what might be possible.)

Although I'm not in deep learning myself (I'm a computer vision researcher), here's a TL;DR as I understand it: rather than having people in specific domains such as computer vision or speech processing create their own features, the idea is to take raw inputs (pixels in the case of images) and train multi-layer neural net architectures that "learn" the relevant higher-level features in an unsupervised way (i.e., without labeled training data). Some of these seem to be pulling out interesting features and perform competitively on a few benchmarks in vision and other fields.

I'm not sold on this yet, because it seems like the complexity of designing features has merely been traded in for the complexity of designing different learning architectures, but it's certainly becoming quite popular these days (mostly led by Geoff Hinton of Toronto, Yoshua Bengio of Montreal, and Yann LeCunn of NYU).

Thanks for that, definitely some prominent folks involved.

Is this the same as non-parametric learning?

No, non-parametric learning is an unrelated topic, which seeks to estimate probability distributions without using "parametric" (= having known structure) models. The canonical example of a non-parametric approach is using histograms, or their generalization, called Parzen Windows.

http://en.wikipedia.org/wiki/Parzen_Windows

That's an orthogonal distinction. The methods thus far have typically been parametric, in that there's a fixed network topology and the learning algorithm adjusts the (fixed set of) weights on the edges. There's no reason, though, why you couldn't have a nonparametric version that adaptively chose the number of hidden nodes in the networks and the connectivity structure.

Also, take a look at some of the work by J.Schmidhuber (http://www.idsia.ch/~juergen/). For example, the LSTM architecture is quite powerful. And he very successfully applied those RNN models to a wide number of areas (handwriting recognition, robotics, ...).

Are you sure about this? I'm not an expert by any means but when I participated in the Netflix Prize a few years, there seemed to be lots of investigation about different variations of neural networks.

For example, the Restricted Boltzmann Machine (http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Bo...), as far as I understand it, seems to be a variation of neural networks.

If you can post a link to an article that covers recent work in the area and explains why they none of them are breakthroughs, I'd love to read it.

The backpropagation neural network itself is a variation of many nonlinear, additive statistical models. The field emerged as an independent entity largely because it had "cool" neurological connotations and was largely ignored by statisticians. The Netflix prize was ultimately won using linear algebra with sensible starting values (the KDD paper that describes the winning method is remarkably easy to read).

backpropagation is just the chain rule of differentiation. each layer of an ANN is logistic regression, which statisticians have been and continue to be interested in.

i'm not sure your caricature of the netflix winning solution is correct: i believe it was a blending of around 25 different models (including, i think, the RBM someone pointed about above) each in themselves quite varied from one another. this is typically how these challenges are won.

A lot of the top teams had blended submissions, but my characterization was not incorrect: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197...

I work on the cinematch team at netflix... I've reviewed all of the winning code... your characterization is incorrect.

The netflix prize would not have been won when it was without rbm.

Is http://www.commendo.at/references/files/kdd08.pdf (this) the paper you are talking about?

Nope, it was by Bell and Koren.

This one is related, but not the one I originally mentioned: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197...

I was at the talk but can't for the life of me remember the title, sorry.

not an expert either, and indeed, the Hinton papers on RBMs are the last thing i remember reading about. But their invention dates back to the 80s. I 'm just noticing that if you search "neural network" on google scholar for the recent years, you will find many neuroscientific results. Of course i dont have the data to back it up.

Well, but he's conditioning on "machine learning" already (the little caption on the histogram says "P(neural network | machine learning"), so that most likely filters out the straight neuroscience papers.

Reply on news.ycombinator.com