> I expect a GPT-5 training cluster to cost $10b (Say 100k blackwell chips + associated infrastructure), and a GPT-6 capable cluster to cost $100b.
You just completely made up those numbers.
You just completely made up those numbers.
GPT-5 being 10x the training cost of GPT-4 is a reasonable estimate. GPT-4 was itself 10x the size of GPT-3. So $10 bil.
GPT-6 being another 10x over GPT-5 is therefore just a extrapolation, hence $100 bil.
It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each.
Today they can buy more powerful hardware and train much larger models for the same cost.
https://nvidianews.nvidia.com/news/nvidia-blackwell-platform...
Just wanted to ask the question - do you think frontier has provided more or less value to the world than gpt-4?
The idea is that you use something like a Frontier to create something like a GPT-4.
My point was about cost of computing. What kind of GPT can you train with a $600M computer?
Why would you estimate that the $100 million training bill would require a billion in GPUs? That's kind of like saying getting a $100 million of water would take a billion in plumbing.
I don't know what the number is, but I'm not going to start just making them up.
Why so much reluctance to even try to make a reasoned guess?
It seems the most ignorant also tend to be the most arrogant.