Hacker News Reader

These days I use FastChat: https://github.com/lm-sys/FastChat

It’s not based on llama.cpp but huggingface transformers but can also run on CPU.

It works well, can be distributed and very conveniently provide the same REST API than OpenAI GPT.

Do you know how well it performs compared to llama.cpp?

my understanding is that the engine used (pytorch transformers library) is still faster than llama.cpp with 100% of layers running on the GPU.

It's the Huggingface transformers library which is implemented in pytorch.

In terms of speed, yes running fp16 will indeed be faster with vanilla gpu setup. However most people are running 4bit quantized versions, and the GPU quantization landscape as been a mess (GPTQ-for-llama project). llama.cpp has taken a totally different approach, and it looks like they are currently able to match native GPU perf via cuBLAS with much less effort and brittleness.

I only have an m1

I don't think the integrated GPU on that supports CUDA. So you will need to use CPU mode only.

Yep, but isn’t there an integrated ML chip that makes it faster than cpu? Or does llama.cpp not use that?

unfortunately that chip is proprietary and undocumented, it's very difficult for open source programs to make use of. I think there is some reverse engineering work being done but it's not complete.

Reply on news.ycombinator.com