Hacker News Reader

Comment by marcopicentini | Hacker News Reader

What do you use to host these models (like Vicuna, Dolly etc) on your own server and expose them using HTTP REST API? Is there an Heroku-like for LLM models?

I am looking for an open source models to do text summarization. Open AI is too expensive for my use case because I need to pass lots of tokens.

View on HN

These days I use FastChat: https://github.com/lm-sys/FastChat

It’s not based on llama.cpp but huggingface transformers but can also run on CPU.

It works well, can be distributed and very conveniently provide the same REST API than OpenAI GPT.

Do you know how well it performs compared to llama.cpp?

my understanding is that the engine used (pytorch transformers library) is still faster than llama.cpp with 100% of layers running on the GPU.

I only have an m1

I don't think the integrated GPU on that supports CUDA. So you will need to use CPU mode only.

Yep, but isn’t there an integrated ML chip that makes it faster than cpu? Or does llama.cpp not use that?

unfortunately that chip is proprietary and undocumented, it's very difficult for open source programs to make use of. I think there is some reverse engineering work being done but it's not complete.

It's the Huggingface transformers library which is implemented in pytorch.

In terms of speed, yes running fp16 will indeed be faster with vanilla gpu setup. However most people are running 4bit quantized versions, and the GPU quantization landscape as been a mess (GPTQ-for-llama project). llama.cpp has taken a totally different approach, and it looks like they are currently able to match native GPU perf via cuBLAS with much less effort and brittleness.

I haven't tried that but https://github.com/abetlen/llama-cpp-python and https://github.com/r2d4/openlm exists

Weights and Biases is good for building/training models and Lambda Labs is a cloud provider for AI workloads. Lambda will only get you up to running the model though. You would still need to overlay some job management on top of that. I've heard Run.AI is good on that front but I haven't tried.

https://bellard.org/ts_server/ may be what you are after.

You can run open-source models, but the software itself is closed-source and free for non-commercial use.

Reply on news.ycombinator.com