Hacker News Reader

Comment by goethes_kind | Hacker News Reader

I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically.

So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent?

So then what is the label for each layer? Do you use the same label for each layer?

And what does he mean by the forward pass not being fully known? I don't get this application of the blackbox between layers. Why would you want to do that?

View on HN

Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.

Jürgen did it all before in the 80s, however it was never translated to English so Geoffrey could happily reinvent it.

Jürgen invented AGI in the early 90s but someone pressed the red button on his website and it committed suicide.

Just curious, why would one want to avoid citing Schmidhuber's work?

It is a bit of a meme in AI research, as Schmidhuber often claims that he hasn't received the citations that he thinks he deserves.

https://www.urbandictionary.com/define.php?term=schmidhuber

Its not just a meme in this case btw :D https://twitter.com/SchmidhuberAI/status/1605246688939364352

Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of a sphere's (3d) surface are good and outputting what regions of a circle's (2d) perimeter are good. This is why he mentions the need for normalizing vectors otherwise layers would cheat and just look at the vector's magnitude.

The idea is imo similar to how random word embeddings are generated.

> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs.

I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely:

    The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself

[1] https://www.cs.toronto.edu/~hinton/FFA13.pdf

Let's suppose that you are correct, which direction are the weights updated towards?

The implementations of this compute gradients locally.

It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?

In my opinion, yes if and only if the update does not use a stateful optimiser, and the computation is easy / simple enough that the updated parameter value can be computed immediately.

In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters.

This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.

It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.

> because the idea is trivial in hindsight (always is) so publishing fast is important.

Unfortunately I've also seen papers get rejected because their idea was "trivial", yet no one had thought of it before. Hinton has an edge here though.

Perhaps a better place to find algorithmic details is this related paper, also with Hinton as a co-author, which implements similar ideas in more standard networks:

Scaling Forward Gradient With Local Losses Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton https://arxiv.org/abs/2210.03310

and has code: https://github.com/google-research/google-research/tree/mast...

> There's only one equation

Not accurate for the version another commenter linked: https://www.cs.toronto.edu/~hinton/FFA13.pdf

I see four equations.

Reply on news.ycombinator.com