Do you have a few of these cutting-edge algorithmic advances papers in mind, could you list them?
I guess I got too pessimistic because of things like "emergent features" [1] / "grokking" [2] that seems to happen only with a lot of compute, and also the fact that the original (vanilla) transformer architecture remains (one of) the best, despite many additional ideas and "advances" (but that is only evident at large scale) [3].
Because of the points above, it's really hard for me, as a non-expert, to assess which papers are true advancements, and which were only published in pursuit of vanity metrics (e.g. publication counts) but actually represent overfit/cherry-picked results rather than robust progress.
[1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea...
[2] https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanis...