Hacker News Reader

Comment by Aurornis | Hacker News Reader

> it seems really far off from the kind of experience even a basic $20/month subscription gets me.

The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.

The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.

But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.

View on HN

There are a few use cases that are (somewhat) surprisingly unsuited for cloud providers:

- translations: cloud providers can bowdlerize (censor) bad words/content; also, if you want to do a translation for personal use of copyrighted materials, cloud providers may block it

- image generation: generating drawings with a style that even just resembles a copyrighted one (ie. Disney) may be blocked by cloud providers - for example, generating old cartoons style with GPT may not be possible.

What about a light but bulky AI job, like batch processing 50GB of files? I'm currently doing it on my used Macbook M1 Max 64gb, and it's chugging through it for the cost of electricity (free with my solar).

64gb unified memory is incredibly powerful for a local machine. Most of us have 16GB which is almost useless compared to the API models.

Right now it's about €1650 for that machine (used, on ebay obviously)

Cloud does not mean censored. You can rent gpu time and run whatever model you want, with your data kept as private as any other cloud instance you personally run. Cloud is location, with (for some work) wayyyy cheaper access.

Cloud is still not "your computer" so it's probably wise to take that into account and act in accordance with your own threat model.

Uncensored models are also popular reasons, although it’s more of a niche.

Aren't those also commonly available from cloud providers?

i'd say 'with' instead of 'from', the cloud provider rents you the hardware, what you run on it, is whatever you show up with

I don't think it is really about "sensitive", but basically about any content you put in. Why would you give corporations your reasoning (data on how you interact with AI, how you "talk" etc.).

All of this is private, but not necessarily sensitive. You never know what is happening with this data. They might say they don't log it or don't sell it, then few years later you'll find it all online or read a book that has a story eerily similar to what you chatted about with GPT a year ago.

Although in this use case, it's likely because GPT guided you to write the same story as somebody else. Talking with an LLM about an idea is a great way to make it more predictable and homogenized. If you're fixing a bike or writing software, this is usually a good thing.

Gemini not long ago, when you said something "useful" said thank you, I will use it to help other users with similar problem. When asked "why would you do that, I thought our chat is private?" it would respond "Apologies. My mistake, of course this chat is private and your information will not be used." Funny.

> much stronger than the local models you can run

but depending on what you're doing, you may not need the "bleeding edge" performance

Banks, Biglaw, and the Pentagon all do it in the cloud. What could an individual be working on that is so secretive?

> Banks, Biglaw, and the Pentagon all do it in the cloud.

In _a_ cloud: their own virtual private cloud. They also have enough power to negotiate contracts with strong privacy provisions.

> They also have enough power to negotiate contracts with strong privacy provisions.

What privacy provisions would you want to add to AWS? Most of the reasonable strong privacy provisions you'd want are already there and/or available if you want to sign up for it, even including US Govt Top Secret data if you meet some approval.

If they decide to ignore the contract and violate the privacy provisions, the US government can punish them, can the company from future contracts, throw executive in jail.

If they violate their privacy agreement with me there's effectively no punishment I can get that they will actually feel. There would have to be a class action law suit and they always just settle those for some small amount and admit to no wrong doing

> If they decide to ignore the contract and violate the privacy provisions, the US government can punish them, can the company from future contracts, throw executive in jail.

To be honest, they'll just send a few million and some gold plated trinkets to the right people and it'll go away. You know that's how things work now in the US. Even if you're an actual convicted drug dealer you can have your mother say a few nice things and whoop there's your get out of jail free card. This stuff is peanuts compared to that.

If not they'll just stall in court and settle for peanuts after everyone is exhausted battling it out. That's what they've always done before the current government made it even easier. Fines are not a penalty, just a business expense.

Nobody important is going to do any time for this. Not in this America.

This kind of stuff is available off the shelf at any major cloud provider.

Guaranteed privacy is available off the shelf?

Why is the security in your home lab so much better than Amazons? Be specific.

Yes, insofar as any "guaranteed" is cloud providers just saying "we pinky swear to not decrypt your resources!"

I know what I'm talking about here - trust me when I say that the difference between a "government" or "large corpo" account and "some dude" is literally a few flags to prevent resources being created in the wrong regions. It's nothing that a random customer can't do themselves.

I like to buy specific brand of soap. I don't want them to know that, it is my right and so is running local LLM "wasting" money on local inference to keep track of my stack of soap.

Those companies also have data sharing/use agreements that they can get from Cloud AI providers due to their size and spend. The secrecy and data protection is largely what they are paying for. Those types of agreements just aren’t available to individual customers. It’s only when you’re spending $$$ that it becomes worth it for the provider.

The two issues I have with relying on the cloud for intelligence is:

- I don't trust them with my data

- I don't trust the data they send back to me

The first is about my privacy, the second is about not being manipulated by what is considered the politically correct answer to my questions.

glm5.3 is matching fable in lots of places and beating it after more than 1 pass in many. now of course when i say this, folks would claim that it's not local, but it can be. if you happen to own a mac studio 512gb, you could run it.

It also takes some load off the AI data centers.

IDK if that might be a concern for Apple or their AI partners.

It worsens the supply crunch, no? A unit you use sparingly vs that memory going into a GPU that serves many more people.

Those will use different wafers, so unless that memory is allocated for unified memory vs gpu HBM it won't make a difference.

What supply crunch? Tons of RAM available for purchase. It's just expensive. That there is a "supply crunch" is made up to benefit from Trump administration not giving a shit how big corps operate

There's cloud hosts out there with unused compute. Wasted cycles are all over businesses running unused cloud apps and subscribed to services they don't use.

Still need a local computer to access the cloud; so a barely used gadget still exists. And this creates duplication of effort; we built RAM for servers AND the edge devices.

Seems redundant when tech nerds and corporations are really the only people that care.

And all that data in the web is meaningless yet we create a supply crunch storing it in servers.

This an out of touch nickel and dime perspective given the big picture to say nothing of the mess of strip mining and manufacturing pipelines that go into every screw, wire, and such

Surely this is not something that motivates the vast majority of people using local LLMs.

Reply on news.ycombinator.com