The cost of developing GPT-2 and GPT-3 was on the order of millions of dollars, well within the budget of most tech organizations. See https://arxiv.org/abs/2005.14165 for the total compute invested in them. OpenAI had raised only a few tens of millions in donations at the time, as a non-profit organization.

The increase in performance of computation has been happening for so many decades now that it's been given names like "Moore's Law." People like Hans Moravec predicted way back in the 1980's that the cost of compute would continue to decline and become cheap enough for AGI by the 2020's or 2030's. That's half a century ago!