Yup. I found this article quite enlightening:

https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...

Prediction as an objective basically forces the models to model the casual processes that create the text itself. It's not going to stop getting better unless the data is insufficient/unvaried or the architecture creates a bottleneck.

I think by the time the former is an "issue", we'll have a Super Intelligence on our hands anyway.

The latter is looking less and less likely to be a real hurdle. Very little inductive bias to steer away from crucial solutions, very scalable.

The TinyLlama project is trying to do that pushing by training a small 1.1 billion-parameter model on 3 trillion tokens: https://github.com/jzhang38/TinyLlama