Hacker News Reader

Comment by lern_too_spel | Hacker News Reader

The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682

The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.

View on HN

Google certainly had influential papers on language models having emergent behaviors, but this isn't the one that inspired scaling up GPT. It was published August 2022 and GPTs at OpenAI were getting scaled up long before this.

From the GPT-2 paper: ``Also thanks to all the Googlers who helped us with training infrastructure.''

This paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.

I don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.

Maybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”

Then they're not "emergent"

I do not think that word means what you think it means.

Can't tell if you're responding to me or parent. But that's what I was gonna say to @canjobear...

Reply on news.ycombinator.com