Hacker News Reader

I haven’t looked at the structure carefully, but It’s hard to guess there are shared layers between models. Likely the input layers for sure, since there is no need to tokenize separately for each model (unless different models have specialized vocabulary).

This is my understanding as well. Also includes the parameters of the expert-routing gating network.

mixture of experts gates on the feed forward network only. the shared weights are the KQV projections for the attention mechanism of each layer.

Explanation from Andrej karpathy makes sense on why: ''' "8x7B" name is a bit misleading because it is not all 7B params that are being 8x'd, only the FeedForward blocks in the Transformer are 8x'd, everything else stays the same. Hence also why total number of params is not 56B but only 46.7B. '''