"the reference code is in the default FP32, and given a tolerance threshold (1e-02)"
that's a huge tolerance and allows them to use fp16 operations to replace the "fp32" kernel.
that's a huge tolerance and allows them to use fp16 operations to replace the "fp32" kernel.
Replacing float32 operations with float16 is also pointless. There is nothing to be gained by doing this, as it removes the actual accuracy advantage of float32s, which would the single most important reason to use that version of the algorithm.
I think this error is large enough that referring to it as FP32 is misleading.
Also, the performance gains do not translate to my RTX 3060M GPU (3.8 GFLOPS vs PyTorch's 5.3), presumably because it lacks the optimized hardware for half precision.
But on the plus side, the single file was very easy to adapt and the code is quite readable. I have seen much uglier kernels.
maybe this intuition is wrong but would be great for the work to address it explicitly if so!
Very likely a kernel for a standard library could not employ such a trick that relies on alignment of input pointers. Certainly not without a fallback.