It's the same line of thought that gave us jumbo refrigerators, muscle cars and supersize meals.
In the American mind, bigger = better.
That that's not necessarily the best way to better results has been demonstrated by the Deepseek shock.
I do wonder if more foundational research would be a better investment.
If you have the best foundational research, but then find yourself blocked by compute, that’s a bad place to be in.
At the moment compute is a constraint too (at least for the smaller shops, maybe not OpenAI!)
We don’t necessarily know that the next AI breakthrough will need less compute either! A breakthrough that gives 5x performance for 20x training compute and 2x inference cost would still be a giant breakthrough and need a lot of hardware! Although this sort of breakthrough is unlikely, and it’s likely to be smaller steps forward - but I think it might be the case that models continue to get more computationally expensive even if there are breakthroughs that make models more efficient because we will want higher capability.
Put another way - I doubt OpenAI would have built GPT-3 faster if they had less compute.
So you're saying a bigger amount of research is better?