Summary: The latest batch of language models can be much smaller yet achieve GPT-3 like performance by being able to query a database or search the web for information. A key indication is that building larger and larger models is not the only way to improve performance.
Hope you find it useful. All feedback is welcome!
Parameter count is not very accurate measure of model performance. Mixture of Expert models like the Switch Transformer [2] can be 1 trillion parameters in size, but are not 5X the performance, for example.
They clock the retrieval at 10 ms, unclear if that includes the BERT inference, however. My assumption is that it does not.
Any plans to do one on MOE for LM?
Thank you!