People at Microsoft Research certainly do. But the bet is that, with scale, these limitations won't matter in practice. For example Transformers can't recognize well-nested brackets (like {[()]}) to infinite depth; the depth that Transformers can recognize is limited by how many self-attention layers they have. But in practice, you rarely need much depth.