My team would like to do training on long text documents to do analysis on them, but they can be thousands of words long.
My guess is that the RWA model would be well suited for natural language processing. I am biased. Of course, you would need to first convert each word into a vector representation (i.e. word2vec). You might also want to add a decay term in the weighted average to force the model to focus on the recent past and not the "deep" past.
(where z and a are z(x_i, h_(i-1)), etc)
The model I used is: h_t = f( (numerator_(t-1) + z x e^a) / (denominator_(t-1) + e^a) )
Equations (7) in the arxiv link (here it is again: https://arxiv.org/pdf/1703.01253.pdf) provide the update equations. When implementing the equations, you sometimes have to scale both the numerator and denominator back by a constant factor to avoid an overflow error