My wish is they would move on to the next phase. The whole deal with SSMs look really good. But looking for better architects is countered with "a regular architecture with more parameters are doing better. What's the point of this"
Furthermore, I think a replacement will require that we _understand_ what the current crop of models are doing mechanically. Some of it was motivated in [1].
[1] https://openaipublic.blob.core.windows.net/neuron-explainer/...
Personally, I think going linear instead of quadratic for a core operation that a system needs to do is by definition an optimization.
I think it's dominance is not going to substantially change any time soon. Dont you know, the solution to all leetcode interviews is a hash table?
There are certainly tradeoffs to both, the general transformer motif scales very well on a number of axis, so that may be the dominant algorithm for a while to come, though almost certainly it will change and evolve as time goes along (and who knows? something else may come along as well <3 :')))) ).
My bet will be on something else than gradient descent and backprop but really I don't wish any company or country to reach agi or any sophisticated ai ...