Hacker News Reader

Comment by cjauvin | Hacker News Reader

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones), would you care to elaborate?

View on HN

I think this post means the same as this tweet:

> Transformers are a special case of Graph Neural Networks. This may be obvious to some.

https://twitter.com/OriolVinyalsML/status/123378359362695168...

It depends on what kind of understanding you want to achieve. It can be helpful to think of DNN as approximating the corresponding infinitely-wide versions. Depending on how you deal with certain scaling, they then act like a linear filter of the error signal in function space, or for single-hidden-layer networks at least, an interacting particle system. In both cases you can understand the convergence of gradient descent training using these analogies, although gaps from real-world practice exist.

To get the neural network to do something useful you need to formalize some way of training it too?

While they can be thought of as stacked regression, it's only logistic regression with one particular non-linearity. And for many non-linearities you'll have a hard time usefully interpreting them as a regression.

I think the common ones have statistical interpretations (that predate deep learning by a lot). Perhaps the one for the rectified linear unit is pretty obscure. But as I understand it, the statistics concept is called the "Tobit" model. It's meaning is not so obscure though, just a prediction that can be non-negative only, which is pretty common like a mass or energy.

Same here, I’m not sure what he’s trying to say.

Reply on news.ycombinator.com