Hacker News Reader

Jan is a good starting point. Desktop app that will guide you through downloading models you can run locally. Then

A similar tool, local desktop app, is LMStudio. Same deal.

Thanks so much! Any resources for deploying something like this into a cloud/datacenter and standing up an API?

They both can be told via command line to run headless and start a server that is OpenAI compatible and can receive documents. Can be run in docker. The API modes are nevertheless not intended for concurrent use as a true server. Another project called VLLM, can be found on github, is intended to support concurrent OpenAI compatible API service, multiplexing over the available GPU.

How expensive is it to run these in the cloud? Last time I looked into this it looked like you have to pay significantly in server costs to be able to run a local LLM.

Reply on news.ycombinator.com