Nice
What is the size of the database and how does it compare to the size of the gpt3 model? "( 2 trillion multi-lingual tokens )" But how much memory is that
How much compute does each method need?
Can you run it on a laptop?
What is the size of the database and how does it compare to the size of the gpt3 model? "( 2 trillion multi-lingual tokens )" But how much memory is that
How much compute does each method need?
Can you run it on a laptop?
Compute: Building the database requires lots of BERT pre-computation. But at inference time, RETRO is at least one BERT forward pass (a batch of all the chunks that it broke the input prompt into). Then the neighbors are computer via the Retro encoder (which seems small at 2 Transformer layers). Then the input prompt is processed similar to a GPT of 32 layers (with attention to the neighbors).
Run on a laptop? Perhaps on CPU. Try running T0 [1] which is larger at 11B parameters.