Hacker News Reader

Comment by innagadadavida | Hacker News Reader

This is a little harsh. Hinton trudged along with neural networks through the coldest AI winter and helped create the conditions for OpenAI to have all the raw ingredients needed to cook up something powerful.

View on HN

If you need to build an airplane, would you rather consult Newton, the Wright brothers, or a modern aerospace engineer? Inventing a field and snatching up the low hanging fruits doesn't mean somebody would be able to consistently create leading edge output. Most of the advances in deep learning are due to hardware scaling, and the success of a few very specific architectures. Yes credit's due where credit's due, but academia name recognition is very much winner take all. For all the criticism Schumidhuber has received, he has a point. The authors of Attention is all you need, the transformers paper, yolo, have nowhere close to the name recognition of the Turing award trio despite generating similar if not more value through their ideas.

> The authors of Attention is all you need, the transformers paper, y

Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before:

https://arxiv.org/abs/2102.11174

Schmidhuber claims to have invented a lot of things. It's almost a running gag at this point.

And yet somehow his claims always bear some truth. I understand the comments about boys crying wolf, but it's hard to ignore the facts on the ground.

not having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?

Often it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.

"Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).

There are multiple dimensions here - fame and fortune at the very least and whether it is localized or global in scope.

It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)

Reply on news.ycombinator.com