> especially not an hypothetical open source organization like discussed here

Hypothetical open source organization is not hypothetical: https://www.eleuther.ai/home

However, the compute constraints are correct. We plan on using TFRC's TPUs in order to replicate GPT-2 (with a few changes to support scaling to GPT-3 and beyond as well as the larger context length). With success on a GPT-2 model we hope to garner enough interest to get more compute to train a full-sized GPT-3 model. After that, it's 1T or bust.