I have to admit ignorance. Is this just a parameters race? Like is 3T parameters 6 times better that 500b? Does the task / work scale linearly with parameters? I know I could just ask chatgpt but wanted to support human content on the internet.
The performance does scale up with parameters, though it’s not linear.
As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget.
So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.
I've heard conflicting things and I'm curious to hear your reasoning!
Loss= (N0/N)^a+(D0/D)^b
Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.
Loss = A/parameters + B/dataset_size + C
Just read the chinchilla paper.
See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682
Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728
All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).
What more corpus is even available? If the corpus is a matter of human annotation, what exactly is being annotated in say the Illiad? A tweet that says, "You up?". The lyrics to 8 Mile. So on and so forth.
And the frontier after is probably having a fleet of actual robots capture additional information, as per this paper.
The trade off is in the cost for training. But in return, you get a smaller and cheaper to run model that can be used by more people.
That is not what PALM E is about though. This is the paper where they patch an LLM so that it can give command to a real, physical robot and basically let the LLM interact with meatspace.
And neither missiles nor drones are capable of functioning totally independent with both requiring large amounts of weapon systems infrastructure. You may be able to argue missiles could be deployed almost independently if failsafes are circumvented but that would require modernization of missile weapon systems which is about as likely as the colonization of space.
At present I’d be more worried about network and social engineering attacks using LLM which I’m sure is already happening and will continuously increase.
Now, we could see another model architecture than the current reigning transformer architecture upend this (much work is ongoing on breaking the quadratic term in the transformer that computationally bounds its performance - an example is the Hyena paper that was published just the other day).
Biggest computer and most data wins is still the paradigm here.
Speaking very generally, it's not just the number of weights ("parameters") but how you use them. The architecture of the neural network is where the most interesting new work is taking place.
Stable Diffusion, which was beating out Dall-E2, was a big deal not just because the weights were public, but also because they were small enough to run on consumer hardware. This is because of the clever architectural choice (plus some other methods.) This was surprising to me, because most of the interesting image generation required huge models running on commercial GPUs.
You'd generally expect diminishing returns from adding weights (as well as a lot more training time and maybe more data) but this is not a law.
I know I could just ask chatgpt but wanted to support human content on the internet.