Code for Scaling Transformer to 1M tokens and beyond with RMT (arxiv.org)

I found this interesting link in paper, so this can be used as ready code for RMT Transformer

https://github.com/booydar/t5-experiments/tree/scaling-repor...

Finetuning:

https://github.com/booydar/t5-experiments/blob/scaling-repor...