How would you respond to the argument that the size of the database should be accounted for when computing the size of the total model? How does the latency of the database lookup compare to the extra latency from running the full size gpt3?
Parameter count is not very accurate measure of model performance. Mixture of Expert models like the Switch Transformer [2] can be 1 trillion parameters in size, but are not 5X the performance, for example.
They clock the retrieval at 10 ms, unclear if that includes the BERT inference, however. My assumption is that it does not.