Maddness is their more recent work and yields 100x speedups: https://arxiv.org/pdf/2106.10860.pdf
The code for Maddness is in the same github repo if you search for "Mithral".
SIMD instructions can work wonders in the right context.
The code for Maddness is in the same github repo if you search for "Mithral".
SIMD instructions can work wonders in the right context.
Also it looks like the optimization is related to running operations on a compressed representation, for the 10x vs 100x speedup, is there a tradeoff between speed and accuracy, or is that extra degree of magnitude just from bringing SIMD into the picture?
Back-of-the-envelope calculation suggests that this won't beat tensor cores on NVIDIA GPUs. This is basically because ~half the die is an ASIC for dense (and 2:4 sparse) matmuls, with no support for the sparsity structure we induce. If 1:16 sparsity were supported or there were a batched warp_shuffle instruction, we'd get similar speedups for GPUs as we do on CPUs.