AI Scaling Myths
aisnakeoil.com
aisnakeoil.com
I don’t see this as good evidence that model parameter increases have reached a limit.
I see it as evidence that compute costs are very high.
GPT-3 cost 4.6 million dollars to train. According to a cursory Google search, GPT-4 cost 100 million dollars to train, about 50x as much. Is a 0.3x improvement worth 50x the cost?
That doesn't even touch on the costs of inference. Millions of queries a day requiring heavy duty compute infrastructure. How much capital are we going to waste on the next marginal improvement? Perhaps Microsoft will "only" need another 30% increase in their carbon emissions this year to sustain the growth.
I find gpt-4 and gpt-3 a world apart. Perhaps it's just that it crossed a tipping point that made it useful for _me_ and the next 10x investment in training wouldn't achieve the same effect for me (but perhaps for somebody else?)
There are many things which earlier models maybe somewhat did, but only large models do reliably to the point of being usable for more than tech demos
He also says
>Paradoxically, smaller models require more training to reach the same level of performance.
It's not a paradox at all. If less training to reach the same level of performance was true, that would be a paradox whereby they'd be trained for under a nanosecond to achieve optimal performance/size payoff.
These are largely not problems money can solve.
Economic disparity is systemic, and changing the system would be nearly free if we knew how to change it to alleviate such disparity.
Community well-being has more to do with involvement than money.
Getting away from electronics is not only free, but will save you money. People just choose not to, and money isn't going to fix that.
Schools could use more funding though. I'm with you there.
Hell the effective altruists would say giving a trillion dollars to people would be pretty effective.
We already know how to do that and it has already been tried with great success. The reality though is that humans simply aren't ready for it. You can see the hypocrisy in real socialist countries. They could adopt the system and become the heroes of humanity overnight and yet they decided to oppress their citizens and await the collapse of their poorly thought out system instead. If the people claiming to fight against capitalism don't actually want to fight capitalism then expecting people who like capitalism to end it is even more hopeless.
You said changing the system would be nearly free. In theory it is. You could end capitalism with an investment of less than a billion dollars. Probably twenty million if you can get volunteers on board for ten years and even less if you could do it in a country with less expensive bank regulations. Yet I do not see such things happen. Nobody is actually trying. When you tell people the solution all I see is their inner capitalist who rejects the idea because they didn't actually care about their current life being shit, what they wanted is for the remote chance of being rich occurring to them to be worth the sacrifice of a life in poverty to keep them complacent. They would never choose a two times better life if it meant giving up becoming rich. Both the communists and capitalists are willingly harming themselves and making their lives worse and they accept that. It is utterly pointless, but here we are. There is a lever you could flick to end it, but they don't.
In a way the economists are correct. The problem is that people aren't selfish enough.
Cash is not coal. We're not burning it to get AGI. People will be getting paid to build and run the hardware.
This is a cash grab from the infrastructure players. Always been.