There are actually neural networks with explicit optimization layers but I don’t think these have really had much success.
Afaik the most important benefit of transformers aren't their “performance” (in the sense of ability to perform their tasks) but their scalability which come from their ability to be trained and evaluated efficiently on big GPU clusters, which isn't something you can do with recurrent neural networks.
And then, if I understood correctly, the benefit of state-space models being that you can train them in parallel and run them in a recurrent fashion, making inference cheaper than transformers especially when context size grow.
It was also my understanding that without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. We needed both very large models and words put into semantic context in semantic space.
Getting rid of RNN vastly improved training scalability and allowed big players to start training enormous models on even more enormous training set in ways that weren't possible with a RNN AFAIK.
You're right that getting rid of "Recurrence" was another innovation, but removing it was probably more of a hack to make things parallelizable, than something that was architecturally justifiable from first principles (like self-attention is), because there's definite "power" in Recurrence (making it desirable), but it's just too costly to run that in LLMs because of CPU cycles.
But that's the entire point of it. Transformer-based LLM are “more intelligent” just because you can make them bigger and train them on bigger datasets because of this parallelization.
SSM are literally the proof that all that really matters is training scalability.
The Universal approximation theorem doesn't care about the architecture after all.
Take our "current large size" (my words from last post) LLMs, as they are currently today, and then simply remove the Self-Attention wiring, and see if that destroys the emergent intelligence aspect or not. I claim it would. But at the same time this doesn't mean you can just stick Self-Attention onto a small model and expect intelligence to once again emerge.
Also, performance of the modern “small” models show that your last sentence isn't really true either.
How could I be "overestimating" the emergent capabilities when I never even quantified those capabilities other than to call them "emergent" and impressive?
> “small” models show that your last sentence isn't true either.
I never said that even a perfect architecture would make small models "intelligent". However to the extent that even smaller LLMs can exhibit surprising capabilities, that's more evidence IN FAVOR OF everything I've said, not evidence against.
EDIT: But in that last sentence (of prior reply) by "small" what I meant was genuinely small, meaning non-LLM, and you seem to have interpreted it as "a smaller LLM"
“All” that was needed to get there was “just” feeding it more data. The fact that we were actually able to train billion parameters models on multiple trillion tokens is the key property of the transformers, there's no magic beyond that (it's already cool enough though): it's not so much that they are more intelligent, it's simply that with them we can brute-force in a scalable fashion.
If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any.
There aren't many multi-billion-parameters non-transformer models because of path dependence, but that doesn't mean that only transformers can achieve this kind of results.
Your position was that the Self-Attention is a less important part (because UAT, yadda yadda), and my position was that it's the key ingredient. Every statement above that I made, that you called wrong, was correct. lol.
You claimed that self attention was needed to achieve the level of intelligence that we've seen with GPT 3.5:
> without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. (Verbatim quote from you https://news.ycombinator.com/item?id=41986010)
This is the claim I've been disputing, by responding that the key feature of the intelligence of tranformer models come from their scalability. And now that we have alternative that scale equally well (SSM and RWKV) unsurprisingly we see them achieve the same level of reasoning abilities.
> Every statement above that I made, that you called wrong, was correct. lol.
Well, except the one quoted above at least…
That attention heads are mandatory for transformers is a tautology (without it a transformer is just an MLP…) so of course this statement is going to be correct, by definition.
But when you move the goal post to land on a tautology then you've surrendered your abilities to argue anything and you are just ridiculing yourself. Take this question of your for instance:
> If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any.
Which is a legit, non-ridiculous, one.
If you replace it with your later much weaker argument:
> > If you know of any MLP that have had success (even at the GPT-2 level), I'd be interested to know what they are, because I don't know of any.
Then it becomes a dumb question given that MLP have no way of encoding context and can't process sequences of words in the first place.
So when you argue that it was your argument all along, it's particularly embarrassing because you're just arguing that your previous arguments were equally dumb even when they weren't.
That's why I said you're disrespecting your earlier argumentation by retreating to your later tautology.
> It's perfectly legit to discuss how a Transformer would perform if only the Self-Attention part was removed
It only shows that you don't understand the topic at all (but hey, you talked about closed-form solutions and quantum computing elsewhere in this discussion with others so why I am even surprised…)
> Insofar as the actual other networks you've mentioned they fail to beat Transformers
They don't “fail to beat transformers”, they beat transformers that aren't the state of the art and are less good that the ones that are. And that's not really a surprise given that they are more recent and have much less manpower working on them. I don't expect them to replace transformers until they make some hypothetic breakthrough that'd makes them significantly better than transformers. That's what path dependence is. But they are still a good illustration to the point that you don't need to have attention heads to exhibit the capabilities of LLMs. (Remember you set the bar at GPT-2 level, and they are far beyond that)
> because language comprehension simply cannot be done without sensitivity to word context
And these models actually have a way to represent context so this criticism completely miss the mark. That's really hilarious that you make this kind of claim in an HN thread about SSM. How come you have no idea at all about what a state-space model is and then feels confident enough to come and argue in the comment section…
Yes, a breakthrough that does what Self-Attention is doing, rather than just scaling up.
But the ship has sailed and nobody is gonna switch to something else than transformers if it's not significantly better, and as such the other approaches are going to stay behind because every marginal improvement come to transformers first (because that's what practically everyone is working on) and alternative models are playing catch-up.
This is a remarkable example of path dependence.
Interpreting this as “transformers are fundamentally superior” is the mistake I'm trying to help you correct.
The breakthrough of transformers was scalability. The next breakthrough of equivalent importance will be entirely different or it won't be.
(English isn't my first language, BTW, so I'd be grateful if you could point the grammar errors)
The 4 words that inadvertently summarized your entire disposition.
You've spent all your credit for any form of consideration at that point.
And now you've even spent your credit for my attention at all, this conversation ends here.