The biggest model that they have used has only 760M parameters, and it outperforms models 1 order of magnitude larger.