Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?
You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$.
You can build a normal pc with a 4090, otherwise you're in datacenter territory.
You can bundle up more GPUs.
You can decrease the precision of your models to get them to run on cards with less RAM.
And could I fit models bigger than 40GB into A100?
Alternatively, the 4000 series cards rely on the PCI express lanes for linking, so if you want to get the benefits of linking the 4000 series cards as a consumer you’ll want a Threadripper or even maybe an EPYC CPU/motherboard combo.
For comparison, an RTX 3090 has 935GB/s of memory bandwidth. As the other person mentioned, that would likely be a much better card if you can find a used one for a reasonable price... but that's just my opinion.