Google Brain outperforms LSTMs with Convolutional Networks in speed and accuracy, seeming to confirm LSTMs are not optimal for NLP at least:

https://arxiv.org/pdf/1706.03762.pdf