This work is quite interesting, and I'm always happy to see such large improvements for memory usage and computation over existing high performant models. I imagine transformer-based models will be with the ML community for a very long time (like ResNets).

I will say that it is somewhat comical that an already existing paper titled "Hierarchical Transformers for Long Document Classification" isn't mentioned in the related work. But to be fair, the two papers are only modestly similar.