Hacker News Reader

Comment by throwanem | Hacker News Reader

This has been coming for a long time and it is why I use local models only. I'm willing to give up capabilities in exchange for being able to trust that whatever biases may exist in the models I do use remain static and predictable.

View on HN

We only have access to local models because they're subsidized too. There's nothing to prevent companies or state actors from paying/funding for increased probability and inherent bias.

Also local models are close in capabilities now but who knows in a few years what that'll look like.

Eh. Files on my hard disks change when I say. And getting hooked on ChatGPT is like - is exactly like - getting addicted to money. If I benefit less from what I do use, I'll accept the trade of never having that rug yanked out from under me. It looks to me like raw model capabilities are topping out anyway; the engineering around them looks like making more difference in the back half of the decade, and I see nothing essential about that requiring nuclear-powered datacenter pyramids or whatever.

> We only have access to local models because they're subsidized too.

Yes, and the flow of future models may dry up, but the current local models we'll have forever.

I agree with this, though i'm still using Claude atm. I figure if we're aware of the downsides you pointed at then we can skip the fast changing landscape of self hosting. It keeps getting cheaper and cheaper to self host, so i'm not sure at what point it makes sense to invest.

For me the switching point will probably be when they (AI companies) start the big rug pull. By then my hope is self hosting will be cheaper, better, easier, etc.

Better not to form the habit, I thought. I'm sure I miss out on some things that way, but that is the lesser risk.

I do use the Gemini assistant that came with this Android, in the same cases and with the same caveats with which I use Siri's fallback web search. As a synthesist of web search results, an LLM isn't half bad, when it doesn't come as a surprise to be hearing from one at least.

Perhaps you should evaluate in terms of the price premium for speed. Sometimes you buy milk at the 7-eleven instead of the grocery store. It costs more, but is worth it for the convenience in the situation you are currently in. Most of the time it is not.

You can buy a used recent PC for a hundred or two, cram it full of memory, and then run a very advanced model. Slowly. But if you are planning to run an agent while you sleep and then review the work in the morning, do you really care if the run time is 4 hours instead of 40 seconds? Most of the time, no. Sometimes, yes.

The difference is not nearly so stark.

How do you do it? Do you host on your hardware or do you use cloud-based providers for open models?

You have a very curious definition of "local" if that includes "cloud-based providers".

> the models I do use remain static and predictable

Some people overload "local" a bit to mean you are hosting the model - whether it's on your computer, on your rack, or on your hetzner instance etc.

But I think parent is referring to the open/static aspect of the models.

If it's hosted by a generic model provider that is serving many users in parallel to reduce the end cost to the user, it's also theoretically a static version of the model... But I could see ad-supported fine tunes being a real problem.

I am bewildered to hear the sense of "local," in which I and engineers in my experience have for thirty years referred to things which are not remotely hosted, referred to as an "overload."

My definition of local is the same as yours. It's just that my mind got focused on the open models part and I forgot that you specifically mentioned a "local" setup. My bad

I thought it was friendlier than "abuse"

Less absurd, anyway. In what sense does one "abuse" a word to use it in its dictionary definition? You appear to be a good example of where this benighted excuse for an industry is headed. Explain it to me.

I assume this means, for example, using their own AWS / EC2 instances to store and process -- not "local" geographically, but "local" personally.

It's a two-year-old Mac Studio in the other room. Where are you coming by this novel sense of 'local?'

Locally it's pretty simple to run models on GPUs, even low powered ones. Have a look at gpt4all as a starting point but there are plenty of offerings in this space.

Not the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access.

Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API.

An M4 Pro would do much better do to the increase in RAM and GPU size.

If you have Docker (Desktop) installed, with just a couple of clicks, you can get a local model going on your computer. llama3.2 (3B), llama3.3 (70B), deepseek-r1, and about a dozen others.

Do you know if there's a site which will help me pick a model to run locally, based on my system specs and needs?

This has been the hardest thing for me to learn and since everything's evolving so quick, what's recommended one week might not be the next.

What does your hardware setup look like?

Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API.

Reply on news.ycombinator.com