So dubious bash completion requiring a cluster of GPUs in a a closed cloud and the energy of a hydroelectric dam to run?
Does it really matter how it’s solved the first time?
The second version can require three orders of magnitude less hardware. Premature optimization is the root of all evil.
Isn't the whole point of GPT-3 to have tons of data and hope for the best by training it with a huge amount of time and energy ?
For training -- using the trained algorithm is cheap. It's a one-time cost (well, one-time in the final version)
You do realize that reducing costs means dropping parameters. So what you are suggestion is reversing the very technique that enables this result. Color me doubtful.
It doesn't. There exist far more sophisticated techniques to reduce the size of a neural network while minimizing loss of accuracy.
The energy required to use GPT-3 is a small fraction of the energy required to train it.
Indeed, the demos of GPT-3 should specifically tackle this.