My somewhat facetious take is that LLMs are trying really hard to reinvent RNNs and would do so if we just gave them the tools to do so.
A different way to think about it is Transformer models are trying to predict which part of the RNN network is "worth" keeping given a resource constraint.
Transformers use a simple heuristic today (and this result makes the heuristic better). Just like many NP complete problems, there might be approximations that are not perfectly correct but still useful. Transformers prove that is the case for neural networks.
What's really needed is a leaderboard based on some private test sets, where you can submit a model to be judged and some entity will run it on their own machines without disclosing the tests to you. Even that could be vulnerable to cheating if the submission process is automated and you can submit many times, as you could use the score as feedback to disclose information about the contents of the private test set. So it would take some care to run such a service, even beyond normal security concerns like securing the sandbox that submitted code runs in, etc.
The existence of models like RWKV indicates that there is potentially a future in training like a transformer but inferring like an RNN.
Many things learned over the last three decades with smaller (the current terminology is "extremely tiny"! :) neural networks are being revisited for these large models.