Transformers.jl and TextAnalysis.jl already provide quite a bit of functionality for NLP, though to my knowledge neither makes use of RNNs. You may be interested in commenting on https://github.com/FluxML/Flux.jl/issues/1678.