A further question that I'd like to add - are there any _truly_ open source initiatives that can remotely compete with GPT-3? What's stopping the OSS community (specifically, not just "funding")? Is it lack of compute power? Lack of data?
Surely, at that budgetary scale, buying/building your own infrastructure would be cheaper than renting it? This is the same economic calculation large animation studios have to make WRT render farms (which also use similar types of hardware).
Almost no one has that kind of money to invest up front (especially not an hypothetical open source organization like discussed here). Even if somehow one had that money available, for this to be more economic than renting you need to have it running at full capacity at all times for years, not many research groups can come up with many interesting projects at a scale large enough to utilize that kind of cluster efficiently.
So buying something like that is only conceivable for huge research groups with hundreds of researchers and tons of cash: Google Brain, DeepMind, Microsoft and probably Facebook. Even OpenAI wasn't able to afford it directly...
Hypothetical open source organization is not hypothetical: https://www.eleuther.ai/home
However, the compute constraints are correct. We plan on using TFRC's TPUs in order to replicate GPT-2 (with a few changes to support scaling to GPT-3 and beyond as well as the larger context length). With success on a GPT-2 model we hope to garner enough interest to get more compute to train a full-sized GPT-3 model. After that, it's 1T or bust.
A trillion parameters? Damn.
Again, I have a feeling that big-budget render farms have trod this path before.
If you throw absolutely enormous amounts of training data and compute resources at a bigass neural network, you get surprisingly good results. Anyone else with a few tens of millions of dollars could do similarly.
It's more a lack of compute power problem plus a lack of people problem. The compute cost alone was estimated to be 4.6 million USD. As for the people, folks who can push SOTA are extremely sought after. So the individuals involved also pay a large opportunity cost by working on things for free or comparatively small open source pay.
Also, I think many open source organizations/communities haven't recognized the value of open source ML models yet and don't invest in them as much as in parametric models. I think this will come with time. Last, anyone with a computer can write code, while few people have access to the hardware needed.