Hacker News Reader

Comment by liuliu | Hacker News Reader

1. I missed this paper https://www.microsoft.com/en-us/research/uploads/prod/2018/0... when doing my own feature map compression research: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de.... Thanks for point it out!

2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick": https://arxiv.org/abs/1404.5997. This scheme is much convoluted but very effective when training large MLP. It is not popular probably because a). the implementation is non-obvious; b). large MLP fall out of fashion quickly, and the computation shape for transformer looks probably very different from the MLP its originally trying to solve (with 4096 activations per layer).

View on HN

This is really intersting.

> In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activity is the unit being communicated), while data parallelism is efficient when the amount of computation per weight is high (because the weight is the unit being communicated).

In this case, for fully connected layers, the amount of computation per neuron is high because it is (fully) connected to and from every other neuron in the pervious and following layer, and therefore we want more GPU's to work on those in parallel, so it isn't a time bottleneck?

What does it mean then for to have a high "computation per weight".

>• Convolutional layers cumulatively contain about 90-95% of the computation, about 5% of the parameters, and have large representations. • Fully-connected layers contain about 5-10% of the computation, about 95% of the parameters, and have small representations.

Wouldn't fc's require more computation since each parameter has more incoming and outgoing values?

I don't doubt this is all accurate and I'm just missing some obvious intuition, but for the sake of learning feel free to explain it if you wish!

Convolutional kernels are matrix-multiplied over and over again in a sliding window across the feature map. The fully connected layer is matrix-multiplied exactly once. So the number of times each element of the weight matrix is used depends on the size of the layer for FC, but depends on the size of the feature map for Conv layers. Usually the feature map is much larger than the kernel, for the majority of layers, so the kernel weights are re-used a lot.

The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.

The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN layers has much more parameters than CNN layers, and all-reduce for gradients on these FCN layers are not economical (BTW, that's why Transformers do mixed model / data parallelism as well, or do large batches, they have a lot parameters and want to do these all-reduce as late as possible (or avoid at all, with model parallelism)).

Otherwise the peer comment answered your question really well!

"One weird trick" is still pretty much the first way to go for most recommendation systems that are large embeddings focused. see torchrec, nvidia's hugectr

Great! Will check out more. Admittedly, recommendation system is one that I have less knowledge about.

Reply on news.ycombinator.com