Google Brain’s Switch Transformer Language Model Packs 1.6T Parameters

Here's the paper: https://arxiv.org/abs/2101.03961