Hacker News Reader

Comment by alok-g | Hacker News Reader

Two questions for the fellow HNers:

1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible?

2. With the amount of memory used jumping up, would there be a significant growth for companies making memories? If so, which ones would be the best positioned?

Thanks.

View on HN

For 1: censorship

A local model will do anything you ask it to, as far as it "knows" about it. It doesn't need to please investors or be afraid of bad press.

LM Studio + a group of select models from huggingface and you can do whatever you want.

For generic coding assistance and knowledge, online services are still better quality.

I don't think there's a huge use-case locally, if you're happy with the subscription cost and privacy. That is, yet. Give it maybe 2 years and someone will probably invent something which local inference would seriously benefit from. I'm anticipating inference for the home appliances (something mac mini form factor that plugs into your router) but that's based on what would make logical sense for consumers, not what consumers would fall for.

Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both.

Thanks.

>> Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both.

So not much Micron? Any US based stocks to invest in? :-)

I forgot about Micron, absolutely. TSMC is the supplier for all of these, so you're covering both memory and compute if that's your strategy (the risk is that US TSMC is over provisioning manufacturing based on the pandemic hardware boom).

Thanks!

IMO it's all about privacy. Perhaps also availability if the main LLM providers start pulling shenanigans but it seems like that's not going to be a huge problem with how many big players are in the space.

I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a client to connect to it and hey presto you've got a secure private LLM everybody in the company can use while keeping data private.

I’ll add to this that while I couldn’t care less about open AI seeing my general coding questions, I wouldn’t run actual important data through ChatGPT.

With a local model, I could toss anything in there. Database query outputs, private keys, stuff like that. This’ll probably become more relevant as we give LLM’s broader use over certain systems.

Like right now I still mostly just type or paste stuff into ChatGPT. But what about when I have a little database copilot that needs to read query results, and maybe even run its own subset of queries like schema checks? Or some open source computer-use type thingy needs to click around in all sorts of places I don’t want openAI going, like my .env or my bash profile? That’s the kinda thing I’d only use a local model for

Hopefully homomorphic encryption can solve this rather than building a new hardware layer everywhere.

Risk of shut-out.

I'm in Hong Kong, I can't even subscribe to OpenAI or Claude directly, though granted this doesn't so much apply to the already "open" models

One important one that I haven't seen mentioned is simply working without an internet connection. It was quite important for me when I was using AI whilst travelling through the countryside, where there is very limited network access.

I don’t currently use AIs, but if I did, they would be local. Simply put: I can’t build my professional career around tools that I do not own.

>> ... around tools that I do not own.

That just may be dependent on how much trust you have on the providers you use. Or do you do your own electricity generation?

That's quite a reductio ad absurdum. No, I don't generate my own electricity (though I could). But I don't use tools for work that can change out from under me at any moment, or that can increase 10x in price on a corporate whim.

And why would that require running AI models locally? You can be in essentially full control by using open source (/open weight) models (DeepSeek etc.) running on exchangable cloud providers that are as replaceable as your electricity provider.

Sure, I guess you can do that as long as you use an open weight model. (Offline support is a nice perk, however.)

We align.

I tend to do the same thing. I do not consider myself as a good representative of an average user though.

In my country things like electricity and water supply are considered a right and a supplier has to go to court to get a supply shut off. Unfortunately we don't yet consider an internet connection in the same way, despite the government essentially requiring it these days.

Even then you're still dependent on someone training new models on sketchy data sources that they don't own.

1. Lower latency for real time tasks e.g. transcription + translation?

Reply on news.ycombinator.com