Hacker News Reader

Comment by tyfon | Hacker News Reader

> Why do these takes around open-source AI remain so popular?

I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs.

The 65B llama I run is actually usable for most of the tasks I would ask chatgpt for (I have premium there but that will lapse this month). The best part is that I never see the "As a large language model I can't do shit" reply.

View on HN

How do you run it locally? llama.cpp + 64GB RAM + 4bit quantized?

I have a 5950x with 64 gb ram and they are quantized to 4 bit yes :)

The weights are stored on a samsung 980 pro so the load time is very fast too. I get about 2 tokens/second with this setup.

edit: forgot to confirm, it is llama.cpp

edit2: I am going to try the FP16 version after easter as I ordered 64 GB of additional ram. But I suspect the speed will be abyssal with the 5950x having to calculate through 120 gb of weights. Hopefully some smart person will come up with a way to allow the GPU to run off system memory via the amd infinity fabric or something.

I thought it needed 64gb of vram. 64gb of ram is easy to obtain

5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.

Quantized models are used aplenty with GPUs as well - 4-bit quantization is the only way you can squeeze llama-30b into 24Gb of VRAM (i.e. RTX 3090 or 4090).

In fact, I would say that, at this point, most people running LLaMA locally are likely using 4-bit quantization regardless of model size and hardware, just to get the most out of the latter.

Most people running llama locally are doing CPU inference, period.

If your desktop had 256gb of ram, could you train a far larger model? Some motherboards support that

How have you managed to run the 65B model? Cloud resources, or you have a very kitted-out homelab?

If you're not running on GPU, you can upgrade your system RAM instead of finding a card with lots of VRAM. 64GB of DDR4 is only $120.

All you need is 2 3090s.

All you need is 64GB of RAM and a CPU, actually. Two 3090s is much faster but not strictly necessary.

All you need is a few thousand dollars lying around to spend solely on your inference fun?

I don’t think that many people really qualify as such (though it’s probably true that many of them are on HN).

Not just inference.

AFAIK, you are able to fine-tune the models with custom data[1], which does not seem to require anything but a GPU with enough VRAM to fit the model in question. I'm looking to get my hands on an RTX 4090 to ingest all of the repair manuals of a certain company and have a chatbot capable of guiding repairs, or at least try to do so. So far doing inference only as well.

[1] https://github.com/tloen/alpaca-lora

you might think about do the training in the cloud and then your back to needing standard hardware for the bot.

Also, another thought might be to generate embeddings for each paragraph of the manual and then index those using Faiss then you generate an embedding of the question and use Faiss to return the most relevant paragraphs feed those into the model with a prompt like "given the following: {paragraphs} \n\n {questions}"

I'm sure there are better prompts but you get the idea.

>All you need is a few thousand dollars lying around to spend solely on your inference fun? I don’t think that many people really qualify as such (though it’s probably true that many of them are on HN).

Can confirm. Did a new build just for inference fun. Expensive, and worth it.

I think it’s hard to verify and those articles get clicks.

Similar to vein of articles promising self driving cars in 202x

Reply on news.ycombinator.com