Curious what analysis you're referencing here.

While I don't doubt people have shown that various transformer models have certain limitations, I'm pretty bullish on transformer models in general.

Here's a post exploring the application of transformers to symbolic mathematics for instance: https://medium.com/analytics-vidhya/solving-differential-equ...