That's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.