Hacker News Reader

~4M$ per full training give or take.

The number thrown around for gpt-3 is $4.6 million, but I am not sure where that figure originates.

It was a number tossed around by a GPU hosting provider, based on their own costs: https://lambdalabs.com/blog/demystifying-gpt-3/

The reality is that GPT-3 was likely "free" to train on Azure, as Microsoft has provided a lot of resources to OpenAI.

If this is true, I wonder what sort of social capital transactional exchange is going on instead.

With training improvements such as DeepSpeed, the GPU costs will likely be substantially lower than what was available at the time OpenAI trained GPT-3. Still not free, though.

The hard part with GPT-3 is it's big enough to make it difficult to actually deploy.

Our current estimate is that it requires between 2000 and 4000 V100 months.