Techniques for Training Large Neural Networks
openai.com
openai.com
2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick": https://arxiv.org/abs/1404.5997. This scheme is much convoluted but very effective when training large MLP. It is not popular probably because a). the implementation is non-obvious; b). large MLP fall out of fashion quickly, and the computation shape for transformer looks probably very different from the MLP its originally trying to solve (with 4096 activations per layer).
> In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activity is the unit being communicated), while data parallelism is efficient when the amount of computation per weight is high (because the weight is the unit being communicated).
In this case, for fully connected layers, the amount of computation per neuron is high because it is (fully) connected to and from every other neuron in the pervious and following layer, and therefore we want more GPU's to work on those in parallel, so it isn't a time bottleneck?
What does it mean then for to have a high "computation per weight".
>• Convolutional layers cumulatively contain about 90-95% of the computation, about 5% of the parameters, and have large representations. • Fully-connected layers contain about 5-10% of the computation, about 95% of the parameters, and have small representations.
Wouldn't fc's require more computation since each parameter has more incoming and outgoing values?
I don't doubt this is all accurate and I'm just missing some obvious intuition, but for the sake of learning feel free to explain it if you wish!
The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.
Otherwise the peer comment answered your question really well!
What’s different?
https://openai.com/blog/evolution-strategies/
Put engineering effort into creating fast, GPU powered (tensorflow/pytorch) algorithms for neuroevolution of the weights of a neural network. I'm still not convinced that properly leveraged gradient free/Feed-forward only optimizers have been tried yet by researchers - namely because they never actually wrote fast gradient free optimizers!
http://pchiusano.github.io/2019-07-06/learning-without-a-gra...
http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
I think you’re right that the area is underexplored.
CMA-ES: https://blog.otoro.net/2017/10/29/visual-evolution-strategie...
ARS: https://arxiv.org/abs/1803.07055
Those optimizers are efficient up to certain neural network sizes.
Source: https://www.eleuther.ai/faq/
What about volunteer-driven distributed computing, like BOINC, Folding@Home, or hivemind? -Backpropagation is dense and sensitive to precision, therefore requiring high-bandwidth communication. Consumer-grade internet connections are wholly insufficient. -Mixture-of-experts-based models tend to significantly underperform monolithic (regular) models for the same number of parameters. -Having enough contributors to outweigh the high overhead is infeasible. -Verifiability and resistance to outside attack are not currently possible without significant additional overhead. In short, doing volunteer-driven distributed compute well for this use case is an unsolved problem.
---
Am really excited to see inroads being made in this field of active research and hope that all AI orgs - OpenAI, Eleuther, etc. - can take part in this in domain of much-needed (IMO) research.
I hope we start seeing more massively parallel training strategies (most likely with GPUs under the hood still)
This is a strange comment to see. The techniques in TFA have been at the heart of a the last few years of progress in large-scale language models, image generation, etc?
Parallel processing as in SIMD architecture is what makes Deep Learning possible now.
Parallel computing as in stacking graphics card is how literally everything is done everywhere- from big tech companies to not-so-well-funded data science departments in unknown unis.
You are so so off.