> The intelligence of an AI model roughly equals the log of the resources used to train and run it.
Pretty sure how log curves work is you add exponentially more inputs to get linear increases in outputs. That would mean it's going to get wildly more difficult and expensive to get each additional marginal gain, and that's what we're seeing.
Essentially the claim is that it gets exponentially cheaper at the same rate the logarithmic resource requirements go up. Which leads to point 3 - linear growth in intelligence is expected over the next few years and AGI is a certainty.
I feel that's not happening at all. This costs 15x more to run compared to 4o and is marginally better. Perhaps the failing of this prediction is on the cost side but regardless it's a failing of the prediction for linear growth in intelligence. Would another 15x resources lead to another marginal gain? Repeat? At that rate of gain we could turn all the resources in the world to this and not hit anywhere near AGI.
4o has been very heavily optimized though. I'd say a more apples-to-apples comparison would be to original gpt-4, which was $30/$60 to 4.5's $75/$150. Still a significant difference, but not quite so stark.
Third point, I think, is that the intelligent work that the models can do will increase exponentially because they get cheaper and cheaper to operate as the get more and more capable.
So I think GPT4.5 is in the short term more expensive (they have to pay their training bill) but eventually this being the new floor is just ratcheting toward the nerd singularity or whatever.
When scaling saturates, less computationally expensive models would benefit more.
We now have llama 3.3 70B, which by most metrics outperforms the 405B model without further scaling, so it’s been my assumption that scaling is dead. Other innovations in training are taking the lead. Higher volumes of low quality data aren’t moving the needle.
Linear increases in intelligence yield exponential increases in economic gains.
Therefore, exponential increases in inputs yield exponential increases in economic gains.
Left as an exercise for the reader is to determine who will capture most of those economic gains.