“Emergent” abilities in LLMs actually develop gradually and predictably – study

There are several issues with the study:

1. Replacing pass/fail accuracy with smoother alternatives (e.g token edit distance) could be a terrible proxy for skill, depending on the task.

2. Even by the authors' metrics, they _still_ find a few potentially emergent abilities.

3. Hindsight is 20-20. Yes, we can revisit the data and fiddle until we find transforms that erase emergence from aptitude plots. The fact is, folk used commonplace test accuracy measurements, and the results were unpredictable and surprising. That's the true notable phenomenon.

I think there's value in the paper. Just...don't take its conclusions too far.

> They find that with a different measuring stick, emergence vanishes.

Isn’t that the case for most/all emergent behavior? If you change the scale and watch individual water molecules, you’d see them snapping into a crystal structure one by one instead of a sudden emergent block of ice.

The paper: Are Emergent Abilities of Large Language Models a Mirage? https://arxiv.org/abs/2304.15004

It's always easier to predict the future after it's arrived.

There may be some emergent features that represent phase changes that are truly difficult, even in retrospect, to predict, but I doubt these are common (does anyone even have a single convincing LLM example?).

I think the more normal case is that higher level capabilities are dependent on multiple lower level ones in ways that may be hard to predict. These smooth improvements in the constituent building blocks may often be there, but you'd have to know where to look (i.e. what the critical building blocks are).

In order to predict emergent capabilities you'd need to have identified ahead of time what the necessary building blocks are, and maybe done some simulation of what level each of those need would need to be at, to support the anticipated/hoped-for behavior.

Of course it's not just about scaling model size and/or data, but also about type and quality of data, where there may be abrupt changes between model versions. It's going to be very difficult ahead of time to analyze what new patterns/manipulations (i.e. building-block capabilities) the model will be able to learn from an updated training set, and therefore what emergent abilities can be built out of those new blocks.

I wonder how often the opposite happens, and model designers have been able to successfully identify that "in order to do X, we need capabilities A, B and C, and in order to get capabilities A, B and C, we need new datasets P & Q" ? In a hypothetical case like that they'd have been able to measure progress towards capability X.

> But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped. Other studies found similar leaps in ability.

Wow, is the title of the submission inaccurate or what!

The partial credit approach is fine, but if we want to train a model to get the right answer, then it does matter.

What I've noticed is that my loss curves on small models for arithmetic will reach a steady state where they're still getting the answer wrong even if some digits are right. Yes you can keep training but the number of training epochs necessary seems to be inversely correlated exponentially with the model size

So a model with x parameters is going to take n^2 time as long as a model with 2x parameters

At a certain parameter count this basically means it is nigh impossible to get the right answer via training using gradient descent

The more parameters, the easier it is to drive to convergence, which is a real, important metric

At some point the estimated time for the ability to emerge spontaneously becomes greater than a human lifetime of even the lifetime of all humanity. So in the sense that increasing model size makes it tractable... I think it's fair to say that the ability emerges suddenly enough.

This is a good paper. Though emergence doesn’t necessarily require a sudden jump in metrics, or unpredictability. New abilities can emerge gradually.

So the models might be getting smarter without discontinuous jumps. And it could just be that we're measuring them in a way that gives no credit for partial answers, thus we end up missing the signs that they've been getting continuously sharper all along.

This sounds kind of in line with what I gather Sam Altman's thinking is. They feel they can predict a model's reasoning abilities quite well based just on the size of the training compute and data.

Even a stopped clock gives the right time, twice a day.

LLMs are plausibility engines. The fundamental hypothesis on trial here is that increasing plausibility corresponds to increasing correctness. This is easily rejected for the human-sourced content that trains them, implying an upper bound on any dependent phenomenon. It follows that simply scaling LLMs will not produce AGI.

The metric the authors use confuses me.

Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`?

Why not consider how far off the model is `abs(actual-expected)`? I wonder if there is an inflection point with that metric.

https://arxiv.org/abs/2206.07682

It's really interesting question how can we measure emergent abilities like arithmetic operations. We cannot test every operation on every possible combinations of numbers. Instead we must make sure somehow that LLM performing arithmetic operations using corresponding rules and axioms.

So an instance of epistemological emergence instead of ontological emergence.

> But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped.

Second paragraph completely contradicts the title given.

This is very insightful. Another one of those "obvious in retrospect" ideas. We tend to think of addition as only correct or not but has anyone ever claimed that LLMs have a "discrete" output (might be using the wrong terminology)? It would then make sense that you need to measure performance in a continuous, not discrete way. Otherwise you'll end up with a sort of "aliasing"-type error.

Okay, neat. Using a binary metric means that you observe sudden transitions from success to failure. Using a more granular metric means you observe smoother improvement. Logical and meaningful.

It does make sense to evaluate things like 3-digit-sum by “how close textually?” And “how close numerically?” and the phase change is an artifact of the actual question being “is fully correct?”

Too bad the article only mentions addition – which shows gradual improvement.

Would have loved to see the claim of emergence supported by an example as well. And more importantly, if the measure is boolean, completely wrong or completely right, then, well, I do expect to see a "sudden" jump in ability.

I think the graph is especially interesting. If the progress of LLMs is less radical when measured correctly, that could be another factor driving a bubble and its later popping.

Transformers have in-context meta learning. If you give them examples they can perform new tasks.

When you run AutoGPT, the outcome is not always predictable.

Phase changes are a natural pattern in complex systems. See state transition in fluids.

Depending on how you measure, they look instant or gradual

I think the importance of this study is being downplayed by many. That same spontaneous and "breakthrough" nature of the abilities of LLM that is now nullified, was a major component of many arguments against LLMs and could've possibly stopped us from achieving greater intelligence.

I see this as the removal of a possible great filter event of the evolution of LLMs. This is a pretty big concern being nullified.

I think it only feels like it is emergent just like how some people thinks LLMs are conscious.

It’s emergent because you don’t really understand what it was capable of? Or is it you have a confirmation bias because you don’t say its an emergent flaw when LLMs get some stuff so crazy wrong

Study: scientists prefer metrics which create an illusion that every time-consuming process is gradual and predictable

Emergence is a weird topic. If we go with the physicist's common definition [0,1] we'd need to differentiate into weak and strong.

As for weak emergence[2] that is de facto.

As for strong emergence[3] well... do we have an example of the phenomena anywhere?

To be clear, ML people have a different definition which I do not think is entirely useful. Let's look at the wording used in [4]

  Scaling up language models has been shown to predictably improve performance and sample

efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence raises the question of whether additional scaling could potentially further expand the range of capabilities of language models

  Emergence is when quantitative changes in a system result in qualitative changes in behavior.
  
  An ability is emergent if it is not present in smaller models but is present in larger models.

I'm not sure anything here is actually meaningful. We have very little knowledge about the interpretation of ML models, so it seems to be jumping the gun to say that phenomena cannot be predicted. The aspects are highly coupled with things like training techniques, optimization objectives (or loss functions), optimization methods, not just architectural designs.

We've also after more research found smaller capable of performing things that large models can do. So the goal post moves and makes the condition of emergent abilities. [RTFA]

So how do we define emergent abilities? Everything static except for the number of parameters? Does this make sense? Changing the number of parameters is an abstracted way of changing the optimization technique or training method. Clearly these are also different by nature of different batch sizes and/or a LR scheduler. So it is hard to be consistent.

Not to mention that we are not great at predicting much of what networks can do in general. There's plenty of people working on the subject (though this is a small proportion of total ML researchers, even if we exclude those that are highly application focused). So are we gonna call something emergent if it is just something we don't know about? And who gets to be the one to decide? I've seen abilities called emergent that are clearly a result of training methods like KL Divergence but surprising to individuals who do not have a rigorous statistics and/or metric theory background.

It just all seems to be jumping the gun. It's a new field and a new science. I think it is okay if we're willing to admit that our understanding just isn't great yet. There's no need to embellish or over attribute. These models are without a doubt powerful and useful, but at the same time I feel we are happy to greatly exaggerate all aspects about them. And for the life of me, I can't figure out why critiques are interpreted as dismissive. You can call an LLM a stochastic parrot and still think it is a great achievement and useful tool. Moreso, criticism is essential. Criticism gives us direction in research. Hype gives us motivation. But the two have to be in balance. Motivation without direction is an undirected Monte Carlo search (can still work, but we can do better than a drunken man). Overly criticizing (turning into dismissing, such as calling LLMs useless) is just pulling wool over one's eyes. It is the equivalent of lights being turned off and deciding to sit down and give up. When the lights get turned off you should find how to turn them back on. And we have more clues than a pure random process. After all, aren't we trying to make these better? I know I am. It is why I got interested in researching these things in the first place. I just wish we could have moderate hype rather than this ridiculously excessive one (which I think operates in a feedback loop with excessive criticism as it drives polarization).

[0] Sabine Hossenfelder (I know HN loves her): https://www.youtube.com/watch?v=bJE6-VTdbjw

[1] Sean Carrol: https://www.youtube.com/watch?v=0_PdLja-eGQ

Yes, Sabine and Sean are both controversial characters but they are well known.

[2] Weak emergence is where some larger phenomena forms out of smaller phenomena and results in something that the smaller thing can't do. The result requires the interactions of parts of a system. Temperature is a common example because it is generally easier to discuss the aggregated value of all the particles' "jigglyness" rather than each individual particle. The resultant property can be derived from the individual components. Conway's game of life is another common example. But in game theory we'd call a coalition an emergent property since utility is higher in the collective than the sum of each individual. Clearly this is true for ANNs since they are composed of neurons and a single neuron cannot perform these tasks.

[3] Strong convergence is about behavior that CANNOT be derived from the individual constituents.

[4] https://arxiv.org/abs/2206.07682

Small pet-peeve with title: Everything is math/predictable. We just may or may not be able to predict it with current knowledge.

OP Should link to the original article which is listed in the first line:

https://www.quantamagazine.org/how-quickly-do-large-language...

That is a terrible headline. It implies that the _abilities_ are a mirage, but it's actually the "emergent" part that might be a mirage -- which is to say it's not an unpredictable "phase transition" but gradual and predictable improvement in ability as the models scale.

There seems to be some sort of flaw or misalignment in the hypothesis. By my reading, it seems this trio is saying LLMs do arithmetic by predicting the digits, and at intermediate stages there are actually partial predictions, and so there isn't a huge jump, which would then be classifiable as emergent. But I'm thinking that in order for an LLM to generally predict arithmetic results in the way they are designed, there would be an infinite memory requirement as it'd have to "store" an infinite set of completions.

So the key may be to find sets of operands that GPT4/Claude Opus are unable to solve, even with hints similar to what a teacher may give a student learning the topic. Until such a set is found, I'd say this capability continues to meet the "emergent" definition, as the only other explanation - that I can think of - is that the models "discovered" how to do arithmetic in a general way.