a fun fact is Bing AI already gives pretty good answers on the tasks Bloomberg cites as motivation, like who is the CEO of various companies, "write a bloomberg excel blp formula to return industry subgroup of ibm apple microsoft google"

another fun fact from the paper, it trained for 1.3m GPU hours and didn't get through the whole corpus, did 80% of 1 epoch (p. 16).

another fun fact, according to Twitter rumors GPT-5 is already training on 25,000 A100s w/8 GPUs each, so 4.8m GPU/hours per day.

it's a good paper with a lot more detail than OpenAI is releasing. also confirms OpenAI is quite a bit ahead. They compare with benchmarks they ran with Bloom and open source GPT-NeoX which they were able to run on same data, and reported benchmarks for GPT-3 since you can't run it yourself (well I guess you could sample the API), and BloombergGPT was competitive with GPT-3 which is a couple of years old now.

my takeaway was, this stuff is hard and OpenAI is crushing it.

I wonder if they will just use it for terminal features, or make the API available, and at what cost, and if they will license the model and weights for people who want to run it internally and maybe train incrementally on their own data. (not sure how feasible that is)