I was going to recommend an A100, but they are not available cheaply anymore. I set someone up with two of them I managed to score from Amazon for like $400 each, but used. They had 40gb RAM, but it was HBM and super fast. There were mining cards based off of these as well, but I have no idea what was stripped down to make them cheaper. The only issue is that you need to heat sink them if they are not going in a server, and in a rack-mounted server, there needs to be proper cooling (big dog fans).

But I just looked again, and A100's are not around at reasonable prices anymore. I cannot even find the old mining equivalents (they used to be everywhere, and CHEAP). Perhaps many people are building similar systems now. After the Ether merge it was a great time to build.

Yes, just ensure you can keep them cool. I have an old m40 that opened up my options with 24GB of VRAM, installed on a traditional case with a 3d printed cooler adapter + fan. While it isn't always fun getting older cards to work, it is certainly viable (and scalable if needed).