Hacker News Reader

Have any of the "sub-quadratic transformers" [1] gone mainstream? Or is everyone simply rich enough to buy enough GPUs.

I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pyt...