FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

github.com

30 points·lnyan··3 comments

It seems to me transformers are here to stay for a while (considering they have been invented in 2017 and there really havent been any fundamental adjustments to the architecture). It is quite exciting to me to think about all the possible optimizations and improvements that can happen if the underlying aritechture stays. I'm thinking about GPU optimizations like this maybe also integration with databases, libraries to simplify interacting and building on top of transformers reducing their sice, fast inference easy domain adjustment etc. etc. Feels like we are at the beginning of a transformer ecosystem.

I agree they're probably here to stay. It's interesting to think about how the hardware-software feedback loop affects architecture choice and research directions. I'm almost certain there is an even more impressive architecture out there, waiting to be discovered. However, until we maximize the performance we get out of Transformers, there is a strong incentive to optimize both software and hardware specialized to this architecture.

One downside is that as hardware/software becomes more specialized to one architecture, it becomes more difficult to explore very different architectures.

To me this is something big, because of it's success on Path-X.

It's a bit surprising that just a longer range transformer architecture enabled by better use of the GPU was the solution though.

Reply on news.ycombinator.com