rohany

> I always assumed that when one warp waits for results from a long latency instruction, another warp, potentially from another block can be scheduled in.

Yes, that is correct. However, most MMA-style kernels that utilize the Tensor Core usually need enough resources per block that only 1 block fits on each SM.

Author here! I think that warp specialization is inherently related to multi-stage pipelining, they aren't really alternatives of each other. Warp specialization is a way to realize a multi-stage pipeline in the face of hazards that may cause the pipeline to spill out of the register file or not let parts of the pipeline run concurrently as desired.

The fact that we tend to need different warp specialization strategies for different hardware is a consequence of the capabilities of that hardware (i.e. different asynchronous instruction types), and contributes to the complexity of targeting that new hardware.

Legion has been used to develop distributed, drop-in replacements for libraries like NumPy and SciPy Sparse -- https://github.com/nv-legate/cunumeric/, https://github.com/nv-legate/legate.sparse

I agree! Much of this work was done as part of the overarching TACO project (https://github.com/tensor-compiler/taco), in an attempt to distribute sparse tensor computations (https://rohany.github.io/publications/sc2022-spdistal.pdf). MLIR recently (~mid 2022) began implementing the ideas from TACO into a "sparse tensor" dialect, so perhaps some of these ideas could make it into there. I'm working with MLIR these days, and if I could re-do the project now I would probably utilize and targetb the MLIR linalg infrastructure!

I am the author of this paper -- kind of crazy to see it posted here on its own! I'll hang around to try and answer questions

Enums should be coming in 20.2 as well :)