They do work, but transformers work better on the accuracy/compute tradeoff curve. Transformers let far away positions in the sequence exchange info in 1 step, while a dilated CNN will need many layers. Additionally, Transformers can be fairly efficiently trained.

I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.