Hacker News Reader

13B in 4bit runs on your average 8GB VRAM consumer video card at fast enough speeds.

The GP's info is outdated. Apparently they don't realize how quickly LLaMA is being developed on. It's just like the early days of SD.

View on HN

llama.cpp runs on the CPU anyways, the whole conversation is on a false premise. The only hard thing about running it on consumer hardware is the quantization step (where the largest RAM requirements come in) and that's not even a technical problem since you could just redistribute the quantized version if it weren't for the license of the model.

Maybe out-dated by 1 day LOL.

Reply on news.ycombinator.com