check out MultiFiT [0] from fastai, it uses QRNN for speed.

[0] http://nlp.fast.ai/classification/2019/09/10/multifit.html