Hacker News Reader

Comment by squeaky-clean | Hacker News Reader

> someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small

These already exist, they're just disproportionately expensive compared to consumer cards. You can get a Nvidia A100 with 80GB VRAM on eBay for about $14,000.

3.33x the memory for 9x the price.

View on HN

You can also buy used enterprise cards on ebay. There may not be many 32gb+ cards up yet, but they'll be on there eventually. The big issue is how to cool these cards though, both amd and nvidia make the cards with fan-less heat sinks because the servers they get packed into provide cooling via their loud, ultra fast, potentially digit severing fans, but people who are salvaging these cards have to find their own solution to cool them, and fit them in a given case.

How purchasers get drivers, don't they require a subscription, or are they pirated?

As far as AMD drivers are concerned, you can at least reflash some cards. Nvidia cards... well I've only seen videos where people say they have a given driver so there might be underhanded means involved.

I'm surprised nobody's yet making custom coolers for them (complete coolers with heatsink/fans, not gluing a 3d-printed fan shroud to the front) and re-selling them. Seems like a niche gap in the market.

Or you could get A6000 for $4k with 48 GB VRAM.

Is 48GB enough to run 70B models ?

Yes, with 4 bit quantization.

Reply on news.ycombinator.com