Hacker News Reader

Comment by omarhaneef | Hacker News Reader

For those who are interested in how Transformers seem more prevalent, please read this thread by Karpathy where he talks about a consolidation in ML:

https://twitter.com/karpathy/status/1468370605229547522

And of course one of the early classic papers in the field, as a bonus:

https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd05...

(The paper is mentioned in the article)

View on HN

if one prefers video, Yannic Kilcher does an excellent explanation of the seminal paper https://www.youtube.com/watch?v=iDulhoQ2pro

Glad to see you here lucidrains. Truly appreciate your recent open-source contributions and works like big sleep, deep daze.

Everyone else check out this https://github.com/lucidrains?tab=repositories

Thanks for the kind words, and credit goes to Ryan Murdock for Big Sleep and Deep Daze. I simply packaged it up to spread the usage

+1, lucidrains is practically a one-person ML start-up

> the approaches were completely different, often not even ML based.

Oh, the horror! Some aproaches were not (gasp!) even ML-based. As someone who was been working all these years in image processing without recourse to much ML stuff, I find this attitude cute and endearing.

I interpreted the “even” there as in contrast to current times.

Do you have some pointers to interesting research of the kind you are doing? Btw, when we're talking about "non-ML" we meen no-learning, not just no-neural-nets, correct?

Thanks in advance.

> Do you have some pointers to interesting research of the kind you are doing?

I don't think my research is particularly interesting for a general public as it is quite niche (low level image processing). Still, we consistently receive industrial funding so it is probably useful to some extent. Some nice journals where we publish our research: Journal of Mathematical Imaging and Vision, SIAM Journal on Imaging Sciences.

> when we're talking about "non-ML" we mean no-learning, not just no-neural-nets, correct?

Why do you say "just" ? Machine learning and neural networks are independent things. There's plenty of wonderful research about neural networks that has nothing to do with learning.

Thanks, I'll look up the journals you suggest.

By "just" I meant to clarify the use of "ML". It's often used to mean "deep learning" and while I thought you didn't use it that way I wanted to make sure.

I normally think of neural nets as a primarily connectionist approach to machine learning (although not necessarily statistical: the first artificial neuron was propositional logic-based). I'm curious to read research that treats them in a different manner. Can you recommend some?

Thanks in advance (again)!

Look for example the article "Approximation Spaces of Deep Neural Networks" by Gribonval et al. It contains a nice review of the spaces of functions that are represented by neural networks. It's just about properties of neural networks and their computing power, depending on their depth, width, skip connections, and nonlinearity type. The weights are fixed constants, there's no training whatsoever.

Thanks!

Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs.

[1]https://www.gwern.net/notes/Attention

I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pyt...

Reply on news.ycombinator.com