> "Training ever bigger convnets and LSTMs on ever bigger datasets gets us closer to Strong AI -- in the same sense that building taller towers gets us closer to the moon." --François Chollet
The Transformer layer has radically leaped over LSTMs and CNNs. While LSTMs can model sequences and CNNs regular grids, they have no efficient long range interaction mechanism. Transformer does. It's a huge leap similar to the one in computer vision from a few years ago.
What is needed besides spatial translation invariance (CNN) and temporal invariance (LSTM) is permutation invariance. Whenever the problem can be described as a graph, then the ordering of the vertices and edges should not matter. You can't do that with CNNs and LSTMs, but you can do it with Graph neural nets and Transformers.
Apparently Transformers are the best for language modelling (GPT-2), playing games (Dota2 from OpenAI), composing music and possibly now in modelling proteins. I assume they will play a huge role in working with graph structured data, with multiple entities and relations.