It has been a pretty painful experience speaking to the companies due to shortages and because of all the limitations of non-H100 chips. For getting a machine with 4 H100s and NVLINK, I was told I'd have to wait one year. With that long of a wait, I might as well wait for the next generation. Without NVLINK, I could potentially get this machine sooner. I struggled to find any reports of the additional complexities and how much slower the machine would be.
I'm basically looking to buy two machines. One for under $40K and one for up to $100K. The $100K machine is intended for training LLMs, and the other would have multiple GPUs where the other would just have a bunch of GPUs for my PhD students to use. For the $40K machine, I'm being told by companies I could only put in 2-3 L40S cards, and they are really pushing these as the only viable option.
In contrast, the last time I had $40K for hardware I got two machines with 4 RTX A5000s and companies were a lot more responsive and seemed more helpful. It is unlikely that I'll have $100K for hardware again, so I'm very reluctant to go with cloud computing since that decreases my hardware budget and I need these to last 3+ years.
I wish I could go with 4090s, but the limited VRAM and that NVIDIA has disabled functionality needed for multi-GPU training makes that pretty much a non-starter.