That opinion was held by a large part of the field for the longest time, and some actually still cling to it. These are usually the people who criticise transformers, because they go against everything they believe. But what we have seen in recent years points to the fact that capability of neural networks is only a question of size. Yes, the human brain uses some tricks like recurrent layers and convolutional layers as well - and to some extent it does so better than we currently can. But transformers have shown that you don't need any of that for language processing and not even for vision, showing once again that you only need a sufficiently sized network. The details of the architecture are not that important. In the same way that your microprocessor architecture does not really matter once you deal with high level programs in userland.