It certaintly was nothing close to half a trillion.
Clearly more money is not the path to the solution. Furthermore China is doing pretty well with a fraction of the spend. America may have money but money needs to go toward productive projects - this requires ideas and vision. Which cannot be bought actually.
Of course any stats undergrad could tell you this was a fairytale. Increasing number of weights only works until you exhaust the signal in the data. I’m not sure how he was allowed to get away with such a blatant lie but here we are.
This lie is effective because on a small time scale it’s impossible to tell the difference between logarithmic growth and logistic growth. If you maintain a fixed training data size, increasing the size of the model will get you logistic growth in model capability meaning that past a certain size you get effectively no gain in performance because you’ve already squeezed out 99% of the signal.
This disproves point number 1 in Sam’s thesis: https://blog.samaltman.com/three-observations
“The intelligence of an AI model roughly equals the log of the resources used to train and run it.”
He is playing loose with the language here because the only way this statement holds is when resources = breadth and depth of training data - not compute / model size.