Ask HN: Transformers is great but what's next?

Transformer architecture is the foundation current paradigm of AI. I agree it's great but what's next, do you think there should have been an innovation since then and another competitor maybe?

It was 2017 when the architecture was proposed and in 2024, 7 years have passed and we are heading towards "Globalization" as Peter Thiel says it.

I would love to know, how far is the next big breakthrough in this Technology that goes from 0 to 1.