Ship something reasonably priced with 32gb+ and I'd be interested!
A 3090.
Ampere will be well supported until its very obsolete because of the A100.
These already exist, they're just disproportionately expensive compared to consumer cards. You can get a Nvidia A100 with 80GB VRAM on eBay for about $14,000.
3.33x the memory for 9x the price.
In Llama V1, 7B was clever but kinda dumb, 13B was still short sighted, but 33B was a big jump to "disturbingly good" territory. 7B/13B finetunes required very specific prompting, but 33B would still give good responses even outside the finetuning format.
16GB is probably perfect for a ~13B model with a very long context and better (5 bit?) quantization, or just partially offloading a ~33B model.
Llama in general is not great for code completion and fact retrieval, from what I have tried
I can't run llama2 70B - but I remember llama1-Guanaco30B was a major leap over the base llama30B model for me.
And you can run this locally with enough combined RAM + VRAM.
First some background: llama is divided into prompt ingestion code, and "layers" for actually generating the tokens.
There are different offloading schemes, but what llama.cpp specifically does is map the layers to different devices. For example, ~7GB of the model could reside on the 8GB GPU, and the other ~9GB would live on the CPU.
During runtime, the prompt is ingested by the GPU all at once (which doesn't take much VRAM), and then for each word, the layers are run sequentially. So the first ~half of a word would run on your GPU, and the last half would run on your CPU, and the alternation repeats till all the words are generated.
The beauty of llama.cpp is that its cpu token generation is (compared to other llama runtimes) extremely fast. So offloading even half or two thirds of a model to a decent CPU is not so bad.
The M1 and M2 Apple processors might change that equation.
The 32gb and 64gb MacBooks can run inference on a lot of these big models.
They might be quite a bit slower in TFLOPS than a 4070, but if they can run the model while the 4070 can't run it due to limited RAM, then they come out the winner
The long and short of it is, right now, CUDA beats TFLOPS.
Now for training, $10K for an M2 with 192GB memory could break even somewhere around a dozen models with current costs, iff the compute is not saturated. The limit of memory bandwidth is only a function of the compute; it doesn't scale infinitely.
The card is 256 bit... Intel could have done this, just like the 16GB RTX 4060 with a 128 bit bus.
Why are gpu vendors deciding how much ram the gpu gets? This feels so 1980s. GenAI workloads have radically variable memory footprints, some folks want 256GB per card so they have a spitting shot at trying something out - and other folks want 8GB per card to maximize throughput.
To add complication, I really don’t want to deal with copying 256GB models between cpu/card. We should have unified memory.
Intel/AMD are reportedly coming out with wide M1-Pro like chips.
Just saw one yesterday, 128GB PCIE 5.0 x8 which is a maximum of 32 GB/s in and out, where the 4090 has 1 TB/s memory bandwidth.
IMO for inference, dual socket EPYC (24 channel DDR5) is the way to go and CXL can theoretically allow you to bump up the bandwidth in that situation (assuming that you can optimize the software properly). Already in llama.cpp there seems to be some issues using 2P servers regarding numa.
...because they make them?
You can go buy 256gb systems if you want, at extreme cost. You can also use an 8gb card for whatever you want too. What are they doing wrong?
> To add complication, I really don’t want to deal with copying 256GB models between cpu/card.
For deployment, it doesn't really matter. You just memory-map it to the CPU or keep it hot-loaded in GPU memory. With a unified memory model you're still limited by your disk speed, so the performance difference probably wouldn't come out great either way.
To my knowledge, this is pretty much how InfiniBand works on datacenter-scale systems like the GH200. The incentive and upsell for bringing this to consumer systems seems weak to me.
The problem is latency. Apple can do what they do in terms of performance because they place the RAM extremely close to the SoC and don't have sockets that present their own RF interference challenges.
If you want to run modular, unified RAM, it gets really messy really fast.
[1] https://www.ifixit.com/Guide/Macbook+Air+(M2+2022)+Logic+Boa...