I doubt that we will hit diminishing returns in AI. We still find new ways to make them faster or cheaper or better or even train themselves...
The flat line prediction is now 2 years old...
The flat line prediction is now 2 years old...
I consider the start of this wave of AI to be approximately the 2017 Google transformer paper and yet transformers didn't really have enough datapoints to look exponential until GPT 3 in 2022.
The following is purely speculation for fun and sparking light-hearted conversation:
My gut feeling is that this generation of models transitioned out of the part of the sigmoid that looks roughly exponential after the introduction of reasoning models.
My prediction is that tranformer-based models will start to enter the phase that asymptotes to flatline in 1-2 years.
I leave open the possibility for a different form of model to emerge that is exponential but I don't believe transformers to be right now.
Sure maybe they do better in some benchmarks, but to me the experience of using LLMs is and has been limited by their tendency to be confidently incorrect which betrays their illusion of intelligence as well as their usefulness. And I don't really see any clear path to getting past this hurdle, I think this may just be about as good as they're gonna get in that regard. Would be great if they prove me wrong.
New and better things are coming. They will just take time to implement, and I doubt they cancel current training runs. So I guess it will take up to a year for the new things to come out
Can the bubble burst in this time, because people lose patience? Of course. But we are far from the end.