Hacker News Reader

The scaling appears linear in relationship, though on some performance metrics it is super linear.

See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682

Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728

All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).

Chinchilla paper proposes the following empirical scaling power law

Loss= (N0/N)^a+(D0/D)^b

Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.

Where A,B,C are empirical constants,

Loss = A/parameters + B/dataset_size + C

Just read the chinchilla paper.

I asked ChatGPT, it said: "Uhhh it depends."