Hacker News Reader

Comment by haolez | Hacker News Reader

How do you do it? Do you host on your hardware or do you use cloud-based providers for open models?

Not the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access.

Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API.

An M4 Pro would do much better do to the increase in RAM and GPU size.

If you have Docker (Desktop) installed, with just a couple of clicks, you can get a local model going on your computer. llama3.2 (3B), llama3.3 (70B), deepseek-r1, and about a dozen others.

Do you know if there's a site which will help me pick a model to run locally, based on my system specs and needs?

This has been the hardest thing for me to learn and since everything's evolving so quick, what's recommended one week might not be the next.

Locally it's pretty simple to run models on GPUs, even low powered ones. Have a look at gpt4all as a starting point but there are plenty of offerings in this space.

You have a very curious definition of "local" if that includes "cloud-based providers".

> the models I do use remain static and predictable

Some people overload "local" a bit to mean you are hosting the model - whether it's on your computer, on your rack, or on your hetzner instance etc.

But I think parent is referring to the open/static aspect of the models.

If it's hosted by a generic model provider that is serving many users in parallel to reduce the end cost to the user, it's also theoretically a static version of the model... But I could see ad-supported fine tunes being a real problem.

I am bewildered to hear the sense of "local," in which I and engineers in my experience have for thirty years referred to things which are not remotely hosted, referred to as an "overload."

My definition of local is the same as yours. It's just that my mind got focused on the open models part and I forgot that you specifically mentioned a "local" setup. My bad

I thought it was friendlier than "abuse"

Less absurd, anyway. In what sense does one "abuse" a word to use it in its dictionary definition? You appear to be a good example of where this benighted excuse for an industry is headed. Explain it to me.

I assume this means, for example, using their own AWS / EC2 instances to store and process -- not "local" geographically, but "local" personally.

It's a two-year-old Mac Studio in the other room. Where are you coming by this novel sense of 'local?'

Reply on news.ycombinator.com