Google breaks the trillion-parameter ceiling with the Switch Transformer

All networks are like this. In a network of N nodes you have N^2 potential relationships. That’s just simple paired relationships. You can still go to higher order groups.

It basically means you need to learn about double descent:

https://twitter.com/hippopedoid/status/1243229024085835779

and regularize your models well, implicitly or explicitly. Your model might otherwise memorize the data instead of learning features.