No More Adam: Learning Rate Scaling at Initialization Is All You Need
arxiv.org
arxiv.org
It seems their method is equivalent to SGD where the learning rate of each tensor is scaled by the number of elements in the tensor. The supposed "Signal-to-Noise ratio" they use is just gSNR=norm(g)/RMS(g-mean(g)), where g is the gradient w.r.t. a d-dimensional tensor and the mean is computed across the elements of g. For a zero-mean iid random gradient the elementwise mean(g)≈0. A similar argument probably holds for arbitrary, but not completely random high-dimensional gradients, mean(g)≈0. In this case gSNR=sqrt(d), which explains why it is constant over time and how it varies across the components of the network.
It also seems the optimal value of their hyperparameter sweeps occurs at the edge in almost every case, and a granularity of 10x for the learning rate and weight decay is too large to make direct comparisons anyway.
Tweaked a bit the hyper parameters and such, but nothing. Probably a bogus implementation?
Maybe it's just way too small, you wouldn't use Karatsuba multiplication to do 3*5.
I'm not using a transformer, just a plain Feedforward, Relu and dropout for a simple classifier.
I don't know, I can be wrong. I hope and some toy experiment shows that even in low case parameters it works fine as well as adam.
They do say it consistently matches or outperforms despite simplicity, and I think that statement means at the lower budget for their approach, but a fair comparison fk seems if it is at least promising would be take advantage of the lower memory read to add more params in their version in the comparison.
Also the paper says slow initial convergence, under limitations:
> More- over, our methods ensure a steady and stable update during training, allowing the model to converge better in a given task with sufficient training steps. Thus, we might observe that the convergence speed is relatively lower than Adam’s in the early stage of training; as our primary focus is to investigate the effectiveness of the SaI approach, we left the acceleration of convergence speed in future work.
My summary is that the memory savings might be great if it works, but it does not work everywhere.
Adam, on the other hand, generally gets you pretty good results without futzing too much with hyper params.
At any rate, by now I'm erring on the side of not promoting or citing them.
After a reread it's nice to see the optimizer is faster but how long is spent in the optimizer and can adamw be tuned for a low memory environment given its greedy to try and reduce the impact of statistical noise on gradient calculations.
Note that when training on image1k it only becomes comparable to adamw after many epoch and infact performs measurably worse for most of the training session. (How significant that is, is up to debate and model/task/data)
Why not incorporate 2nd order changes into adamw directly?
The lower memory footprint is nice but it's not immediately why this is the case. Is the batch size reduced? Model changed? I'll be read this after a 2nd coffee and see if it is more obvious...
Still promising if true.
The problem with adam is that it keeps one more statistic (the same size as the model parameters) in memory. It also adds a little computation.
The way to deal with it otherwise is to tune the momentum-parameters and clip/limit the gradient in various ways.