Hacker News Reader

Comment by atemerev | Hacker News Reader

We want cheap(ish) desktop GPUs with 80G or more VRAM, so not only large corporations could meaningfully participate in AI research.

The price divide between “desktop” and “datacenter” GPUs is artificial and no doubt there is a collusion of some kind.

View on HN

You can get Macs with 192GB of UMA, so maybe 180GBusable VRAM. But of course the GPU horsepower is much less tban an A100.

Sort of works for inference, but not for anything else. For now, I bought a 4x4090 workstation… but I am crying how power-inefficient it is.

For my education, would you be able to expand on what it doesn't work for and why? Thank you!

Inference (token generation) is memory-bound, KV cache prefill (prompt processing) is compute-bound. The ARM Macintoshes have lots of memory bandwidth but not a lot of compute power, so they're great for outputting text but terrible for tasks like analyzing documents. I've never done fine-tuning but my understanding is that that is a highly-parallelizable compute hog as well.

You might like this article, which looks at the arithmetic intensity of LLM processing: https://www.baseten.co/blog/llm-transformer-inference-guide/

You know that their top tier datacenter card has "only" 80GB? Their workstation cards with 48GB are also reasonably priced... with 3-4k pricing.

Nvidia B100 has 192G VRAM.

They are not shipping in any reasonable quantity. They are also 2x96GB.

>Their workstation cards with 48GB are also reasonably priced... with 3-4k pricing.

That's twice the price of a 4090. You can't tell me the extra RAM costs that much.

It's not only VRAM. more cores, more bandwith, better power delivery, workstation features...

Which card do you mean that has more cores/computing performance than 4090 and costs 4-5k?

NVIDIA RTX 6000 Ada Generation

Although they retail for more than $4k.

The previous generation RTX A6000 is probably what they're referring to above.

It is $7.5k, and slower than 4090 compute-wise. Two 4090s (which are cheaper than A6000) are strictly superior to it.

Indeed two 4090s provide more CUDA/RT/Tensor cores, but the A6000 gives you 48GB of RAM which can end up making the card faster at certain tasks that would require swapping out data to system RAM on a 24GB card.

Two 4090s have 48G too. For most (if not all) LLM-related tasks, there is no difference how many cards are used for the same amount of VRAM.

Reply on news.ycombinator.com