Hacker News Reader

Comment by dougmwne | Hacker News Reader

Yes and no.

The performance does scale up with parameters, though it’s not linear.

As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget.

So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.

View on HN

The corpus part is the one that gets me. Presumably Google, Facebook, Microsoft, etc have a mirror of essentially the entire public (and their private) internet communications. An enormous amount of published English literature has been digitized.

What more corpus is even available? If the corpus is a matter of human annotation, what exactly is being annotated in say the Illiad? A tweet that says, "You up?". The lyrics to 8 Mile. So on and so forth.

I would assume that video and audio are the next frontier. Talking about Google, training on the entirety of YouTube would probably be next, if they haven't yet used it.

And the frontier after is probably having a fleet of actual robots capture additional information, as per this paper.

How does that performance scale though? If you double the inputs you train on, do you get twice the performance? Less? More?

I've heard conflicting things and I'm curious to hear your reasoning!

I asked ChatGPT, it said: "Uhhh it depends."

Chinchilla paper proposes the following empirical scaling power law

Loss= (N0/N)^a+(D0/D)^b

Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.

Where A,B,C are empirical constants,

Loss = A/parameters + B/dataset_size + C

Just read the chinchilla paper.

The scaling appears linear in relationship, though on some performance metrics it is super linear.

See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682

Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728

All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).

Reply on news.ycombinator.com