They have been using/building stuff for the LUMI supercomputer which has a bit over 12000 MI250X
https://ir.amd.com/news-events/press-releases/detail/1206/am...
“Silo AI has been a pioneer in scaling large language model training on LUMI, Europe’s fastest supercomputer powered by over 12,000 AMD Instinct MI250X GPUs,”