Hacker News Reader

Comment by rococode | Hacker News Reader

I'm curious, are there papers in other ML fields that could be considered breakthroughs comparable in impact to AlexNet?

For NLP the recent ELMo and BERT papers for word embeddings come to mind, although their scope is somewhat different than AlexNet.

View on HN

Sure, lots! We've made lots of progress just in the last decade or two. LSTM, recurrent, and convolutional nets. Slightly older (late 90's), but I think Random Forests were a pretty significant breakthrough.

Another huge one is Paul Graham's own work using Naive Bayes to filter spam.

Both convolutional and recurrent architectures were developed in late 80s. LSTM (improved RNN) was published in 1997.

Maybe LeNet from LeCun et. al? It was before my time, but the architecture is still influential. The more important paper would probably be Phoneme Recognition Using Time-Delay Neural Networks, which introduces Time-Delay NNs, from which ConvNets arise naturally.

Other architectures, like Sepp Hochreiter; Jürgen Schmidhuber (1997): "Long short-term memory", should also be mentioned.

Not sure which paper would be the most important for SVMs.

I don't know what to do about concepts that took a long time to develop, like backpropagation. Research is usually incremental.

Most research looks more incremental than people think. AlexNet is based on much work by Fukushima, Yann LeCun and others. Likewise, ELMO and BERT are based on 2015 work on pretrained language models:

https://papers.nips.cc/paper/5949-semi-supervised-sequence-l...

Reply on news.ycombinator.com