Comment by visarga | Hacker News Reader

Comment by visarga | Hacker News Reader

Parent Full thread

> We now have the compute, the data, and the architecture (transformer) to do it.

It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. Conversely, humans getting training on the same topics achieve very similar results, even though brains are very different at low level, not even the same number of neurons, not to mention different wiring.

The merit for our current wave is 99% on the training data, its quality and size are the true AI heroes. And it took humanity our whole existence to build up to this training set, it cost "a lot" to explore and discover the concepts we put inside it. A single human, group or even a whole generation of humans would not be able to rediscover it from scratch in a lifetime. Our cultural data is smarter than us individually, it is as smart as humanity as a whole.

One consequence of this insight is that we are probably on an AI plateau. We have used up most organic text. The next step is AI generating its own experiences in the world, but it's going to be a slow grind in many fields where environment feedback is not easy to obtain.

> It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify.

My take is that prediction, however you do it, is the essence of intelligence. In fact, I'd define intelligence as the degree of ability to correctly predict future outcomes based on prior experience.

The ultimate intelligent architecture, for now, is our own cortex, which can be architecturally analyzed as a prediction machine - utilizing masses of perceptual feedback to correct/update predictions of how the perceptual scene, and results of our own actions, will evolve.

With prediction as the basis of intelligence, any model capable of predicting - to varying degrees of success - will be perceived to have a commensurate degree of intelligence. Transformer-based LLMs of course aren't the only possible way to predict, but they do seem significantly better at it than competing approaches such as Mamba or the RNN (LSTM etc) seq2seq approaches that were the direct precursor to the transformer.

I think the reason the transformer architecture is so much better than the alternatives, even if there are alternatives, is down to this specific way it does it - able to create these attention "keys" to query the context, and the ways that multiple attention heads learn to coordinate such as "induction heads" copying data from the context to achieve in-context learning.

If you invented the transformer but didn't have trillions of tokens to train it with, no chatGPT. But if you had Mamba/RWKV/SSSM and trillions of tokens you would have almost the same thing with chatGPT.

The training set is magical. It took humanity a long time to discover all the nifty ideas we have in it. It's the result of many generations of humans working together, using language to share their experience. Intelligence is a social process, even though we like to think about keys and queries, or synapses and neurotransmitters, in fact it is the work of many people that made it possible.

And language is that central medium between all of us, an evolutionary system of ideas, evolving at a much faster rate than biology. Now AI have become language replicators like humans, a new era in the history of language has begun. The same language trains humans and LLMs to achieve similar sets of abilities.

I agree about language - which might be though of as "thought macros". Human experience has taught us what things (objects, actions, etc) are worth labelling, what thought patterns are useful to reason about them, etc. Being able to reason about things in the realm of, and using the patterns of, human language is tremendously powerful.

Are there any Mamba benchmarks that show it matching transformer (GPT, say) benchmark performance for similiar size models and training sets?

I don't think there are Mamba LLMs larger than 2.8B at the moment. But here a crop of papers building on it, mostly vision applications:

https://trendingpapers.com/search?q=mamba

I don’t think we are at a plateau. We may have fed a large amount of text into these models, but when you add up all other kinds of media, images, videos, sound, 3D models, there’s a castle more rich dataset about the world. Sora showed that these models can learn a lot about physics and cause and effect just from video feeds. Once this is all combined together into multimodal mega models then we may be closer to the plateau.

Reply on news.ycombinator.com