However... Ethereum enabled/lured many everyday joes into buying 4+ $1k cards due to the incentive structure of; 'buy mining rig then buy a lambo.' Setting up a GPT-NeoX is orders of magnitude more difficult than pointing GPU compute at Nicehash. I really have a hard time thinking ML will have any meaningful uptake in that regard because the incentive structure isn't the same and its much, much harder.
Big cloud seems to be going there own way in regards to compute. GPU's are great for ML, but that doesn't mean they will always hold the crown. TPU's, NVMe storage, and chiplets may find a better path with newer software.
I just don't see how Nvidia really thrives without drastically reducing price (less margin). I don't think they are dead, but they are in big trouble as are many companies.
StableDiffusion alone was trained on 256 x Nvidia A100 GPUs.
Training is dominated by Nvidia, I will not question that as most papers I have seen say something similar. I will say that I do not believe training will always be dominated by Nvidia's datacenter options. Two things that will hasten the withdraw from Nvidia; Cuda and hardware advances around the motherboard (ASICs, RAM proximity, PCIe lanes, data transfer planes, etc).
Think about this... what if a company released an ML training/Inference ASIC that used regular DDR4/NVMe, performed like 4 x A100's and cost $8000? Would you be interested? I would! I don't think this is too far off, there has to be someone working on this outside of Google, Apple and Meta.