Hacker News Reader

Can you say a bit more about this? The bad tool calling has made me give up on using Gemma for my Hermes and a personal recipe site. I have only downloaded from Ollama.

View on HN

Ollama is not recommended [0], use llama.cpp or more specifically Unsloth Studio which wraps llama.cpp and which has an API mode you can use to hook into Hermes or another agent. Unsloth make both the Studio and the quants which fix various issues with many models [1] as well as implementing new features like MTP and QAT support much sooner than other teams. In general you should read r/LocalLLaMa as it has a lot of updates regarding local models as the field moves fast.

[0] https://sleepingrobots.com/dreams/stop-using-ollama/

[1] https://github.com/unslothai/unsloth/discussions/4921

Everytime I try to mention how shit Ollama is, I get mass downvoted here by folks that don't want to hear the truth: There are 4 good inference engines (okay, 5, but we don't count huggingface because it's slow):

1. vLLM

2. sglang

3. (nvidia only) TRT-LLM

4. llama.cpp (mac only, the above are better for non-mac)

If you're not using one of the above, you're doing it wrong

Reply on news.ycombinator.com