You can't do this kind of math without these multi-million dollar facilities and the government licensing that will eventually be required to run one.
"An award of computer time was provided by the INCITE program. This research also used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725."
Models:
https://www.together.xyz/blog/redpajama-models-v1
Training Data:
But even if what you said was true, this technology is progressing faster than we can write news stories about it. OpenAI itself is only a few years old, and you can already run a GPT-3-class model on your laptop. Any prognostication you make about cost will be valid for approximately a month.
And, I'm sorry I can't help myself, what the heck do you mean by colocated? Exactly what things need to be colocated?
I think he said it was more, and that was for a company already capable of creating GPT-3.5. The only company capable of it, in fact.