The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M.
It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each.
Today they can buy more powerful hardware and train much larger models for the same cost.
https://nvidianews.nvidia.com/news/nvidia-blackwell-platform...