We're trying to run some @PyTorch benchmarks for our open-source inference server NOS (https://github.com/autonomi-ai/nos) and wanted to get a deeper understanding of the state of ROCm compatibility vs. CUDA.
I love this because I predicted this question nearly a year ago now. The answer is that the options today are limited. AMD has previously only focused on deploying these cards to HPC clusters. That said, this is changing quickly.
From what I have read, Azure has some MI300x available on their site.
Lamini (US) has some MI210's and MI250's and one box of MI300x.
ElioVP (Europe) might have some MI2X0's and has one box of MI300x.
TensorWave (US) say they have MI300x, but it isn't confirmed in public.
My startup, Hot Aisle (US), will also have a box of MI300x soon (delayed by Chinese New Year), with significantly more on the way in March/April.
It's just very surprising given that they launched MI210's and MI250's a while back. How is it that AMD is expecting the software ecosystem to mature if the HW is still not available in public clouds for testing/development purposes. I noticed they have their own cloud (aac.amd.com/), but need an invite for access.
Will check out if Azure offers these as general availability.
I agree with you, the developer flywheel is missing today. Part of Hot Aisle's mission will be to work on this over time. From what I've seen, Lamini believes in this aspect as well, as they are more focused on the software side of things. We are initially focused more on being the capex/opex deployment, as these chassis are very expensive and a bit challenging to deploy.
I've used AAC before. It is something that AMD aquihired off another company that developed it. It is pretty snazzy, it easily lets you run a container on their hardware. I doubt it will ever be commercialized by them though. It also has very limited hardware resources as it is always being used. When I used it, the best I could get at the time was MI210's.
Once I get the hardware, I'm expecting to do a bunch of initial benchmarking in order to drum up interest in these cards. If you want to ping me privately, I'll keep you in mind for access. The more software that runs on them, the better.