Five Years of LLM Progress

Ah, Finbarr has the top two slots on HN. That's just below getting a Nobel Prize, in the computer world, at least, I think.

Something to note is this post is from March 2023.

Things move pretty fast. I've moved to my little corner of empirical hyperoptimization, and more theoretical, longer-term projects, as well as some practical ones to make some change, so I don't see nearly as much of the thrash as I used to. I keep up some, and it's crazy how fast it moves.

> Almost every team that I’ve been talking to that is training a LLM right now talks about how they’re training a Chinchilla optimal model, which is remarkable given that basically everything in the LLM space changes every week.

I hope that either that's a miscommunication, or I'm wrong about how much of a red flag that seems to be.

The Chinchilla scaling laws allow you to relate, at a somewhat-better-than-rule-of-thumb level, the model size, training data size, and achieved performance of a LLM, without actually training one. So, if for instance you have a certain loss target, and a certain sized corpus of training data, you can use the scaling law to calculate what size of a model to train to hit the target. I can see that being useful to any team.

Chinchilla-optimality on the other hand means finding, for a set loss target, the combination of model size and training data size that minimizes training compute (which, roughly speaking, scales with just the product of those two numbers). But only training compute: Inference compute only scales with model size, regardless of training data. So Chinchilla-optimality is useful only if you expect training to take up most of your compute, i.e. if you are not expecting to actually use the model that much. I'm not in the field myself so I don't know how to quantify "that much", but it's definitely enough to keep those concepts distinct.

\tangent the early approaches were less efficient than later ones, meaning you need that much more power to get started, so you can start improving it.

Theoretically, the better method could have been worked out from the beginning, but actual progress doesn't seem to happen like that. It may be because a working model enables faster experiments (iteration) to search the space-of-methods faster, and in particular to prune off dead-ends, making the search-space smaller.