If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.
I agree that Mamba doesn't solve everything and it still needs work. But I disagree with the logic that there isn't an issue of railroading.
This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture.
There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total surface area. Neanderthals are estimated to have had larger brains too! It isn't mass, neurons, neuron density, surface area. We aren't just scaled up chimps.
But my point stands - our brains have evolved directly from apes brains and the main difference between them and us is brain size.
>>> What’s the main difference between an ape’s brain and a human brain? Scale.
Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.
When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.
If you really think it is all scale, train a 7T ResNet MLP based model for NLP. If scale is all you need, make a LLM without DPO or RLHF. If scale is all you need, make SD3 with a GAN. Or what about a VAE, Normalizing Flow, HMM? Do it with different optimizers. Do it with gradient free methods. Do it with different loss functions.
The bitter lesson wasn't "scale is all you need." That's just a misinterpretation.
Edit: It's fine to disagree. We can compete on the ideas and methods. That's good for our community. So continue down yours, and I'll continue down mine. All I ask is that since your camp is more popular, you don't block those on my side. If you're right, we'll get to AGI soon. If we're right, we still might. But if we're right and you all block us, we'll get another AI winter in between. If we're right and you all don't block us, we can progress without skipping a beat. Just don't put all your eggs in one basket. It isn't good for anyone.
If you disagree please suggest a better way, or at least provide evidence that scaling up no longer works for transformers.
Isnt the Mixture-of-Experts trend (GPT4 is MoE?) kinda of a proof ?
> I said “scale is all you need for transformers”
No you didn't. What kicked this all off was
> What’s the main difference between an ape’s brain and a human brain? Scale.
Don't retcon.
> An African Elephant has roughly 3x the number of neurons humans have.
An African elephant's brain is not a scaled up chimp brain in any way. African elephants have less cortical neurons than a chimp, and roughly a third of the amount that humans have.
> Dolphins beat humans in total surface area.
Animals even less related to humans and chimps, with no prehensile appendages, living in an environment where building stuff is exceedingly difficult. And of course their brains are obviously different from any great ape.
> Neanderthals are estimated to have had larger brains too!
And were just as smart as us and also had a scaled up chimp brain.
Human :: 16 :: 86
Gorilla :: 9.1 :: 33
Chimp :: 6 :: 22
African Elephant :: 5.6 :: 251
Chimps are generally considered more intelligent than gorillas.
Bottlenose Dolphins have 11-15b cortical neurons while humans are in the range 14-18 (range is measurement uncertainty). It's also worth noting these dolphins have a larger brain mass (1.6 kg) and larger cortical surface (3700 cm2) than humans (1.3 kg and 2400 cm2, respectively).
> with no prehensile appendages, living in...
So more than scale. Glad we agree. Seems you also agree that architecture matters too.
And chimps are genetically more similar to humans than gorillas. A chimp brain is more similar to a human brain than a gorilla brain.
> So more than scale. Glad we agree.
We absolutely do not agree. Notice how nobody suggested that a human brain is a scaled up version of an axolotl's brain? Yeah, that didn't happen. I do wonder why?
What is the best/stable-ish linear alternative for transformer right now? Especially for text generation and summarization.
We have domain specific ways of over sampling and search, so we much prefer less expensive models.
Sure, NVIDIA et al may not want that (although, again I don't see why... they too can't produce chips fast enough so being able to provide models for customers now ought to be good), but there's so much money out there that does...
Simply put, for the time being huge datasets are going to be needed and those with bigger (cleaner?) datasets will have a better behaving model.
- the data/processing having to be large means the data-owners have a benefit
- the data/processing having to be large means NVIDIA also has a benefit (sells more GPUs to handle all that load)
High processing cost is not a benefit to them at all. It's just a cost eating into their margins.