Hacker News Reader

Comment by babl-yc | Hacker News Reader

Agreed -- given the talk is titled "Exciting trends in Machine Learning" it feels pretty incomplete to gloss over ChatGPT blowing up.

OpenAI bet big on (1) emergent behaviors as you scale language models and (2) RLHF / fine-tuning to follow instructions.

Both those topics were lightly covered but very much from the "Google did this" perspective (word2vec, step-by-step reasoning).

View on HN

The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682

The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.

Google certainly had influential papers on language models having emergent behaviors, but this isn't the one that inspired scaling up GPT. It was published August 2022 and GPTs at OpenAI were getting scaled up long before this.

This paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.

I don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.

Maybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”

Then they're not "emergent"

I do not think that word means what you think it means.

Can't tell if you're responding to me or parent. But that's what I was gonna say to @canjobear...

From the GPT-2 paper: ``Also thanks to all the Googlers who helped us with training infrastructure.''

The research breakthroughs all happened at Google, so it's not surprising.

RLHF was invented by OpenAI.

DPO was invented at Stanford.

Exactly. Google.

And all the people that actually did the inventing promptly left Google.

(yes I get there's 2 exceptions)

Reply on news.ycombinator.com