while this topic has visibility, there is also another relatively unexplored research direction, fine-tuning pretrained transformers into RNNs

https://arxiv.org/abs/2210.04243

https://aclanthology.org/2021.emnlp-main.830.pdf