Hacker News Reader

Comment by Eisenstein | Hacker News Reader

The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

View on HN

> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM.

No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model on the GPU is not useful.

> Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

I agree with this, if someone has a desktop that can fit two GPUs.

The main benefit of a GPU in that case is much faster prompt reading. Could be useful for Code Llama cases where you want the model to read a lot of code and then write a line or part of a line.

Multi-GPU in desktop chassis gets crazy pretty quickly. If you don't care about aesthetics and can figure out both the power delivery and PCI-E lane situation, https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... has an example that will make a Bitcoin mining rig look clean.

Water cooling can get you down to 2x slot height, with all of the trouble involved in water cooling. NVIDIA really segmented the market quite well. Gamers hate blower cards, but they are the right physical dimensions to make multi-GPU work well, and they are exclusively on the workstation cards.

> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM.

Then you can just run it entirely on CPU. There is no point to buy an expensive GPU to run LLMs to be bottlenecked by your CPU in the first place. Which is why I do not get so excited with these huge models, as they gain less traction as not as many people can run them locally, and finetuning is probably more costly too.

GGUF is just a file format. The ability to offload some layers to CPU is not specific to it nor to llama.cpp in general - indeed, it was available before llama.cpp was even a thing.

I'm pretty sure I didn't assert that it was more that a file format or that llama.cpp was a pioneer in that regard?

Reply on news.ycombinator.com