Hacker News Reader

Hacker News

Comment by codetrotter | Hacker News Reader

Parent Full thread

codetrotter

> doesn't make a lot of sense (besides privacy...)

Privacy is worth very much though.

View on HN

What privacy benefit do you get running this locally vs renting a baremetal GPU and running it there?

Wouldn't that be much more cost-effective?

Especially when you inevitably want to run a better / different model in the near future that would benefit from different hardware?

You can get similar Tok/sec on a single RTX 4090 - which you can rent for <$1/hr.

But at a totally different quant, you're crazy if you think you can run the entire R1 model on a single 4090, come on man. Apples and oranges.

Definitely but when you can run this in places like Azure with tight contracts it makes little sense except for the ultra paranoid.

Considering the power of three letter agencies in the USA and the complete unhingedness of the new administration, I would not trust anything to a contract.

Sure I am certain there is a possibility but unless you have airgapped your local instance and locked down your local network securely it does not really matter.

It’s cool to run things locally and it will get better as time goes on but for most use cases I don’t find it worth it. Everyone is different and folks that enjoy the idea of local network secure can run it locally.

Even a badly operated on-prem system has the advantage that if someone breaks in, they are taking a risk of getting caught. Whereas with Azure the TLAs could just hoover up everything there without high risk of customers finding out (assuming they can gag MS). Given the reporting about NSA's "collect everything" modus operandi this doesn't seem very far fetched.

hmm do we still have to pretend that this is some sort of conspiracy theory? really? after snowden? it doesn't "seem very far fetched", its a fact

It's less "possibility" and more "certainty."

can we even trust the hardware?

The hardware can be airgapped.

Well you don't need to worry unless you are already on the list.

These days getting on a list may require as little as "is trans" or "has immigrant parents."

or "your competitor donated to Musk/Trump campaign"

no, just being speculative... about that.

Reply on news.ycombinator.com