We do use ROCm and CUDA. Only we sandbox it with the model and download only the needed parts which are about 1/10th of the size.