Does this work on the consumer grade cards like the 7090 XTX?
And by work I don't mean: spend two weeks trying to get the drivers set up and never update the server again.
And by work I don't mean: spend two weeks trying to get the drivers set up and never update the server again.
[1] https://wandb.ai/augmxnt/train-bench/reports/Trainer-perform...
There are also still ongoing optimizations on the Nvidia side as well. In the beginning of the year the 7900 XTX and 3090 were pretty close on llama.cpp inference performance, but a few months ago llama.cpp got CUDA graph and FA support implemented that boosted perf significantly for both my 3090 and 4090.
(For AI/ML, a used 3090 remains I think the best bang/buck for both inference and small training runs. You can pay twice as much for the twice as fast 4090, but at the end of the day you'll still wish you had more VRAM, so it's hard to really recommend unless you're going to use mixed precision. The RDNA3 cards are not as bad to work with as the Internet would have you believe, but they'd have to be a lot cheaper if your main use case was AI/ML for both the PITA factor and just from pure real-world performance.)