Hacker News Reader

Comment by lukevp | Hacker News Reader

I was in the market for a new GPU and I debated between a 4090 and a 4070. 4070 is fine for the type of work and gaming I do, but I was thinking about the additional RAM. Then I realized that the 4070 has more vram than most consumers do, so many models will be optimized for that, and at some point someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small. I don’t know a ton about this stuff so who knows if that was the right decision… but I’m thinking along the same lines as you.

View on HN

> someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small

These already exist, they're just disproportionately expensive compared to consumer cards. You can get a Nvidia A100 with 80GB VRAM on eBay for about $14,000.

3.33x the memory for 9x the price.

You can also buy used enterprise cards on ebay. There may not be many 32gb+ cards up yet, but they'll be on there eventually. The big issue is how to cool these cards though, both amd and nvidia make the cards with fan-less heat sinks because the servers they get packed into provide cooling via their loud, ultra fast, potentially digit severing fans, but people who are salvaging these cards have to find their own solution to cool them, and fit them in a given case.

I'm surprised nobody's yet making custom coolers for them (complete coolers with heatsink/fans, not gluing a 3d-printed fan shroud to the front) and re-selling them. Seems like a niche gap in the market.

How purchasers get drivers, don't they require a subscription, or are they pirated?

As far as AMD drivers are concerned, you can at least reflash some cards. Nvidia cards... well I've only seen videos where people say they have a given driver so there might be underhanded means involved.

Or you could get A6000 for $4k with 48 GB VRAM.

Is 48GB enough to run 70B models ?

Yes, with 4 bit quantization.

Also, 24GB is good because it's just barely big enough for the (currently unreleased) Llama V2 34B.

In Llama V1, 7B was clever but kinda dumb, 13B was still short sighted, but 33B was a big jump to "disturbingly good" territory. 7B/13B finetunes required very specific prompting, but 33B would still give good responses even outside the finetuning format.

16GB is probably perfect for a ~13B model with a very long context and better (5 bit?) quantization, or just partially offloading a ~33B model.

I don't know, maybe I'm desensitized by GPT4 but I'm running Llama2 70B and I certainly wouldn't call it 'disturbingly good'.

Base Llama is not good at all, but the finetunes are quite good at their niches.

Llama in general is not great for code completion and fact retrieval, from what I have tried

Check out the hugging face leader board and try one of the finetunes.

I can't run llama2 70B - but I remember llama1-Guanaco30B was a major leap over the base llama30B model for me.

Thanks! That's useful to know. Unfortunately it looks like a lot of them are non-commercial which rules them out for me but there's a couple that look like they might work.

Its rather outdated. Search for "70b" on huggingface, sort by date, and you will find more cutting edge models.

And you can run this locally with enough combined RAM + VRAM.

Does it have to be exclusively vram to be useful or is shared ram also useful here. I have a system that reports 8gb dedicated to GPU with another 15 that is shared and thus a GPU memory of 23 GB

Driver shared memory is effectively useless. The way it accesses CPU memory is (for now) extremely slow.

First some background: llama is divided into prompt ingestion code, and "layers" for actually generating the tokens.

There are different offloading schemes, but what llama.cpp specifically does is map the layers to different devices. For example, ~7GB of the model could reside on the 8GB GPU, and the other ~9GB would live on the CPU.

During runtime, the prompt is ingested by the GPU all at once (which doesn't take much VRAM), and then for each word, the layers are run sequentially. So the first ~half of a word would run on your GPU, and the last half would run on your CPU, and the alternation repeats till all the words are generated.

The beauty of llama.cpp is that its cpu token generation is (compared to other llama runtimes) extremely fast. So offloading even half or two thirds of a model to a decent CPU is not so bad.

Thank you for the explanation

My compromise:

A 3090.

Ampere will be well supported until its very obsolete because of the A100.

If anything, 24GB is probably the sweet spot for what's optimised for as many local LLM enthusiasts are running 3090s and 4090s (and all want more VRAM).

> Then I realized that the 4070 has more vram than most consumers do

The M1 and M2 Apple processors might change that equation.

The 32gb and 64gb MacBooks can run inference on a lot of these big models.

They might be quite a bit slower in TFLOPS than a 4070, but if they can run the model while the 4070 can't run it due to limited RAM, then they come out the winner

That's true, and you do make a disclaimer about performance, but often when this is brought up, people either haven't actually tried MPS or are unaware that a GPU isn't required for inference.

The long and short of it is, right now, CUDA beats TFLOPS.

Now for training, $10K for an M2 with 192GB memory could break even somewhere around a dozen models with current costs, iff the compute is not saturated. The limit of memory bandwidth is only a function of the compute; it doesn't scale infinitely.

Yes I could run llama 65b on my M1 Max with 64GB of ram.

I do also have a MacBook Pro with 32 gigs of Ram and an M2 Studio with 64 gigs, so if a great model comes out that supports unified memory, I should be good there!

Reply on news.ycombinator.com