Comment by neilv | Hacker News Reader

Comment by neilv | Hacker News Reader

It takes searching and experimenting to figure out what works, and to avoid some of the sketchier stuff (and to lean towards things you could legally use for a startup), but I'm pretty happy with my current home setup, on an old PC with RTX 3090 and 64GB main RAM.

8-bit quantized uncensored Llama 2 13B, doing 50 generated tokens/second, using CPU+GPU including 17GB of 3090's 24GB VRAM.

I also have quantized 70B running currently CPU-only, but I might later be able to speed that up with some CUDA or OpenCL offloading.

This is on Debian Stable (like usual), albeit currently with closed Nvidia CUDA stack, and necessarily with the closed Llama 2 that I can only fine-tune atop. (I'm hoping that some scientific/academic non-profit/govt effort will be able to muster fully open models in the future.)

One of the main reasons I picked Llama 2 was the relatively friendly licensing (and Meta is earning lots of goodwill with that). With this licensing, and the performance I'm getting, in theory, I could even shoestring bootstrap an indie startup with low online LLM demands, from a single consumer hardware box in the proverbial startup garage or kitchen table. (Though I'd try to get affordable cloud compute first.)

I am about to start working on a non-profit project -- not a startup, but similar in terms of resources dedicated to the project and how we hope it will scale.

One of our big questions is whether it makes sense to rent or to buy for training/finetuning/RLHF. The advantage of renting is obvious: I don't think that this phase of the project will last very long, and if it turns out that the idea is a success we'll have no problem securing funding for perma-improvement infra.

The possible advantage of buying is that we would then have the hardware available for inference hosting. We do expect some amount of demand in perpetuity. Having that ongoing cost as small as possible would allow us to continue serving the "clients" we KNOW would benefit a lot from our service with minimal recurring revenue.

Just a suggestion but they have 4bit quantified models that are even smaller and faster that the 8 bit. Your average 13B 4bit model is only about 8-9gb of VRAM. Using this I bet you can get a much higher perimeter model on the 3090.

I was using various 4-bit quantized earlier, but decided to go back to 8-bit for 13B, since I had the VRAM anyway, and (at the time, for other reasons) was seeing some quirky behavior.

70B is currently 4-bit on this box, and once I have GPU accel for 70B, I'll see how the quality compares to 13B 8-bit.

Whoa, 50 tokens/second locally sounds amazing. Any recommendations on guides or documentation for setting up the stack to run on hardware like that?

Reply on news.ycombinator.com