I think it’s hardware not software.

“Cuda moat” is a misnomer. The PTX spec is relatively short (600 page pdf). Triton directly writes PTX, skipping cuda. Flash attention was created by a non nvidia employee without access to any of the secret sauce within Cuda or its libraries.

The hardware is just not as good, and no software can paper over its flaws.