Comment by cr0sh | Hacker News Reader

Comment by cr0sh | Hacker News Reader

Parent Full thread

Note: I am not an expert on ML, ANNs, etc - most of what I have done has been mainly "hobby level" and MOOC learning.

From what I understand, though, what GH is trying to accomplish with capsule networks (which I have tried to understand, and have yet to succeed) is optimizing (or possible removal of) backpropagation.

He has noted in the past how - so far (from what I know) - there isn't a biological equivalent to backpropagation for learning. Backprop is a completely artificial mathematical construct that doesn't happen in natural systems. It is also extremely energy intensive - at least in the manner that it is currently done.

So the question is - that he's hoping to answer I think - what is a proper working framework for implementing an artificial neural network that can learn, without using backpropagation (or using it differently)?

I think whoever can solve this will fundamentally remake the field of ML/AI - much in the same way that backprop (and later "deep learning") did.

It's not the backprop that is holding back ML - it takes about as much time as the forward pass and requires 3x the memory. Backprop is necessary for almost all the deep neural nets that have state of the art results, attempting to replace it would push us back a decade or more.

The main problem with ML is the separation of data and compute. It takes a lot of time and energy to move data around. We need 'in memory compute'.

So you're in disagreement with Hinton?

https://www.axios.com/artificial-intelligence-pioneer-says-w...

I've heard the argument made about "in memory compute" - and I do know that various companies have made hardware toward that direction (and some of it has very low power requirements).

I just find it compelling that Hinton - at least as of 2 years ago - has this viewpoint that we should be rethinking the concept of backprop...

that article is way too vague for agreement/disagreement to be a productive point of discussion imo. is he talking about new optimization methods?

I'd say this is a difference of scale. Hinton's comments are meant to inspire and encourage _The Next Great Thing_. We could read the paper that broadly, but they seem to be addressing a more immediate local minima.

Isn’t backpropagation just an efficient method for getting a gradient? It sounds like what he’s looking for an alternative to is the idea of adjusting parameters to match labeled training data.

Reply on news.ycombinator.com