Hacker News Reader

Comment by naasking | Hacker News Reader

> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs.

I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely:

    The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself

[1] https://www.cs.toronto.edu/~hinton/FFA13.pdf

View on HN

It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.

Let's suppose that you are correct, which direction are the weights updated towards?

The implementations of this compute gradients locally.

It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?

In my opinion, yes if and only if the update does not use a stateful optimiser, and the computation is easy / simple enough that the updated parameter value can be computed immediately.

In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters.

This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.

Reply on news.ycombinator.com