Hacker News Reader

Comment by rottencupcakes | Hacker News Reader

This is dangerous. I think of talking to an LLM like playing a game - it's a complex system and it's fun to mess with it and role play.

Taking what somebody says in an LLM chat as the truth and using it in the real world is a slippery slope. The fears about thought crime are real.

View on HN

I don't think you should think of it like "playing a game", you should think of it as "posting to a web service you don't control". If you posted on a web forum about your plans for violence against the CEO of that forum, they'd probably contact the police too, and I don't particularly think there's anything sinister about this.

So we're creating a system where people who can afford their own hardware can speak to AI freely but the plebes will be monitored by the elites for any attempted uprising?

"Creating"? I don't think there has ever been a system where you could expect to be able to plan an uprising against the elites in public, or using infrastructure those elites own and control. I wouldn't expect to be able to plan a peasant revolt in the lord's manor.

> Richard Stallman was right about everything.

He might be right about this but everything is really pushing it.

In my view, LLM companies are in a tough spot here.

I grew up in Dawson Creek, BC. An hour's drive away is Tumbler Ridge. In February 2026, someone did a mass shooting, killing 6 people and injuring 2 dozen more.

The shooter had previously discussed this with ChatGPT, but OpenAI did not report it to the authorities at the time.

In hindsight, that was an error.

Should the LLM companies be the ones determining if threats are real, or should they just default to letting the police investigate?

I'm fine with the latter. Again, I absolutely believe that these companies have immense power to do evil, I just don't think the linked article is strong evidence of it.

By allowing the monitoring of AI conversations, we are creating a dystopian world where essentially all recorded or spoken thought is monitored (given that AI has or will have access to almost all conversations).

Is this the world we want to live in?

Like anything else online, if you want something to be private, don't post it.

Absolutely nothing you view, send or post on any online service can be assumed to be private.

If someone told you they were planning a mass shooting and wanted advice for how to carry it out, would you report them?

So do you think offline language models should be outlawed or restricted?

I don't know how you inferred that. I said nothing about restricting offline computing.

Ok, so the rules for someone who selfhosts vs. Hosts for others should be different? What if I provide inference services to my family members, should I monitor their interactions and report them if I find anything dangerous?

It's more about what I can expect.

If I keep a diary offline on paper I can expect that to be private.

If I keep a diary on a hosted diary-as-a-service, I cannot.

Many people have different expectations. Isn't that what this whole conversation is about?

You didn't answer my question.

No, I don't want to live in that world, and I would definitely read an article discussing how to ensure we don't reach that world.

That said, I think that world is quite a few steps away from the scenario being discussed, which is "when talking to X, X might report my conversation to the police".

You comment elsewhere that this might be a have vs have-not divide -- if you can afford to self-host a powerful LLM, you will have more liberties.

I think this isn't that trenchant of an observation for two reasons: (1) the wealthy have always had an advantage in exercising their rights over the poor, but more importantly, (2) the capability vs cost ratio of self-hosted LLMs continues to improve at an insanely rapid pace.

... and the advantage of the frontier models over anything you can possibly self-host continues to increase at an even more insane pace.

It's my perspective that open models are closing the gap faster than the frontier is widening it.

I'm not alone in that opinion.

By "open models", do you perhaps mean the approximately trillion-parameter open models that have been getting released recently? Because basically nobody can actually self-host those. And even at that size it's debatable that they're closing the gap.

Talk to me when you can do it with a 24B model quantized to 8 bits.

> Is this the world we want to live in?

Do you think you have a choice? It's Sam, Dario, and their investors' decision about what kind of world they want to build.

The legal fallout from LLMs becoming spy will overwhelm any profit from these misguided attempts at short lived contracts.

> The legal fallout from LLMs becoming spy will overwhelm any profit from these misguided attempts at short lived contracts.

Then the law must be changed to protect the profit.

While I mostly agree, was that really different with googling certain topics, before modern AI?

I had to give this thought, but yes.

I could use Tor or Google from the library if I wanted. Maybe I would accidentally leak my paper trail, like the Dread Pirate Roberts (Ross Ulbricht) did. But it was possible.

The AI systems require an account.

I can still use chatgpt without an account. Maybe it will be a limited model, but for discussing/analyzing some topics it still does fine, most of the time. You just cannot code with it.

The big problem that bothers me is the ability for post-hoc justification.

Any interaction with cops will result in them curating your entire digital presence to craft a damning story about how deviant you are, for doing things like googling TNT

Every time they shoot someone now there's an entire media blitz dedicated to how that person once did something bad.

Reply on news.ycombinator.com