Hacker News Reader

Any plans for being able to run the entire thing locally with local models?

Thanks for asking. We will launch local LLM, STT, and TTS models in the near future versions.

You can make an OpenAI compatible local server using LMStudio[1] and load any model you want. It'd have to be on another host though, the s3 has some inference capabilities with addons afair, but nowhere enough grunt to run locally at any usable token/s

[1] https://lmstudio.ai/

or with open source tools like

llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/...

or

mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/...

lmstudio and ollama use llama.cpp underneath. cut the middle man

Reply on news.ycombinator.com