LLM/state space models have been popular for some years now, see: https://arxiv.org/abs/2212.14052

More recently, hybrid architectures that utilize attention plus other operators are gaining traction.

See https://arxiv.org/abs/2503.01868