Full circle eh. I wonder how well it compares to just trying to use the actual Whisper models on a variety of existing GPU capable bigger frameworks.

I don't know much practically about how hard it would be to take the Whisper PyTorch (1 or 2?) trained models & to make good use of them elsewhere. I expect Whisper.cpp probably better caters to users, is more readily consumable. Doing the same integer quantization that 1.4.0 whisper.cpp does would be another ask: how hard would that be?

Fwiw, Whisper.cpp uses Nvidia's cuBLAS. There does appear to be an AMD rocm port. https://github.com/ROCmSoftwarePlatform/rocBLAS