Hacker News Reader

Comment by 0xbadc0de5 | Hacker News Reader

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig.

Coupled with the leaked Bing prompt and text-generation-webui, the results are quite impressive.

View on HN

Macs have UMA so an off the shelf Mac can use up to about 120GB of vram. Far more than any consumer card, more than the biggest a100.

GPU power is lower, of course, but pure vram is not a problem.

VRAM is the thing that Apple Silicon is going to have in excess compared to anything even close in price. MacBook Airs can have 14-15GB of VRAM if necessary.

RAM read bandwidth of M1/M2 is still not really competitive with the large GPUs like RTX3090, but it's getting close, compared with the usual CPU setups.

Is there an upper limit on the usable VRAM on Airs, or is it just tied to RAM size minus a bit for the OS? Just got a 24GB Air M2 and your comment made me wonder if it was capped at n amount of VRAM, or if it's just that Air's now can have 24GB of ram compared to 16GB with the M1 Airs?

Apple Silicon uses unified memory so laptops have up to 64GB of VRAM and the Max sitio can have up to 128 GB.

That makes them uniquely “powerful” for inference with large models.

Nowadays up to 96GB on the laptops.

Here is some quick math: these devices has SSD read speed somewhere around 2GiB/s. With 4-bit quantization, we are looking at loading 4B parameters per second. That means we need 8s per token for 30B model. Hmmm, the math is a bit off (or I need to look closer whether we can do more tokens per iteration with some batching).

You can use Speculative Sampling, where a draft model is used to generate short continuations of a sequence, which are scored in parallel by the large model. The draft model can be small, and you only need to call the large model from time to time, so you can stream it from SSD or cheap RAM.

Using LRDIMM DDR4 at the price of less than $1000 it is possible to stream GPT-3 five times a second, in 4bit quantisation. Multiply that with the 2-2.5x speedup from Speculative Sampling.

>Accelerating Large Language Model Decoding with Speculative Sampling

https://arxiv.org/abs/2302.01318

I think your math is missing some details.

An RTX 4090 has a memory bandwidth of 1,008 GB/s.

PCIe 4 x16 has a 32GB/s bandwidth.

DDR4 RAM has 3200 MT/s transfer rate.

An AMD 5900x can easily max that out.

A good NVME can hit 7 GB/s read speeds or better.

And the 4-bit CUDA kernels can pack 16x 4-bit ints into a single 64-bit transfer / register.

Reply on news.ycombinator.com