Ask HN: If you're running an LLM locally, what are you using?

5 points·nomilk··4 comments

As in title, if you're self-hosting an LLM, which one are you using and how did you set it up.

For context, I got an image generator running locally in about 20 minutes (Fooocus). Keen to try the same with LLM.

I recently found out about llama-cpp's official "Server" function: https://github.com/ggerganov/llama.cpp/blob/master/examples/...

Works like a charm for my (simple) use case. I'm running it on an Always Free Oracle Ampere A1 instance with 4 cores and ~20gb of memory. (Obligatory "fuck Larry Ellison" here)

I am running ollama on my MSI GP76 windows laptop with RTX 3080 card and 64 gb ram. It's running on the baked in linux installation. It recognized the graphics card right away and works pretty well. On my macbook pro m3 max with 36 GB ram, I can't run the 70B parameter model.

Ollama is incredible for me! The setup was super easy. I have been playing around with llama2-uncensored.

Got it working. Was easy to install on macOS just an .app file to download, and one line in the terminal to download the 3.8GB manifest. Took 44 minutes from making this post (35 minutes from your reply). Seems to work quite well (quite a lot faster than ChatGPT, but perhaps not as high quality, at least based on some of the leaderboards I've been perusing).

I added this web ui mostly to render code chunks nicely for easy copy/paste (web UI found via the readme): https://github.com/ivanfioravanti/chatbot-ollama

Getting chatgpt-like UX. Love that I can put in a system prompt so easily (if desired).

Reply on news.ycombinator.com