The idea is imo similar to how random word embeddings are generated.
The idea is imo similar to how random word embeddings are generated.
Unfortunately I've also seen papers get rejected because their idea was "trivial", yet no one had thought of it before. Hinton has an edge here though.
I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely:
The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself
[1] https://www.cs.toronto.edu/~hinton/FFA13.pdfThe implementations of this compute gradients locally.
In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters.
This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.