Seeing the other comments, it still leaves me the question if this is a limit of scale or a limit of technology? How much more can LLMs scale up over the next decade?
It is also quite possible that there are fundamental limits to scaling LLMs that we have not discovered yet. You won't find a way to let airplanes fly to the moon no matter how many resources you pour into it. To make it there, you need better methods (ie rockets in this case) that look nothing like the first few steps to get off the ground.