Hacker News Reader

Comment by quanticle | Hacker News Reader

Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?

The implication is that the massive gains haven't been the result of any algorithmic breakthroughs, but rather have been due to the application of massive amounts of computational resources, which weren't available when current ML algorithms were invented. So far as I can tell, that's a true accusation. If you look at the papers coming out of Google and Facebook, they talk about throwing thousands of hours of specialized GPU (or even more specialized and expensive TPU) time at some of the problems. The advances have more to do with Moore's Law making brute force feasible than they have to do with algorithmic breakthroughs.

View on HN

A small bone of contention: it’s not about Moore’s law per se, which has been ‘dead’ since about 2013, coincidentally when the deep learning revival started. It’s matrix multiplication ASIC development that is driving the progress.

GPUs already existed when the idea to use them to make the feasible size of neural nets larger came about. For a long time the drive for the increase of GPU compute power was still gaming/commercial graphics houses. It’s really only in the last 1-2 years that we’ve seen highly specialised GPUs with features like tensor cores (or indeed google’s TPUs).

Also, calling neural nets ‘brute force’ because they use a lot of computing power to train a model is slightly reductionist - a true brute force approach to image recognition, ie enumerating all possible combinations of, say, 200x200x256x3 pixels, would be completely absurd and probably exceed the computing power currently available on earth.

I'm not saying that neural nets are brute force. I'm saying that there haven't been any algorithmic improvements in neural nets to make them more computationally feasible than they were when they were first invented. Instead, we have specialized hardware which can just do the necessary computation quickly enough to make neural nets feasible.

It's not like neural nets are a new technology. They've been known since the '80s, at least. It's just that they were considered a dead end, because we didn't have the computational resources to run deep neural nets, nor did we have sufficient training data to make neural-net approaches feasible. Once those preconditions were met, neural nets took off in short order.

There have - convolutional nets have far fewer weights to learn than an equivalent fully connected net of the same depth.

And you literally described it as a brute force approach in your comment.

Reply on news.ycombinator.com