Hacker News Reader

Comment by socializer | Hacker News Reader

I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.

However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.

View on HN

Not just bad headlines; OpenAI is actively being sued [1] for this:

    OpenAI’s stated rationale was a concern for the shooter’s privacy, but its own interests
    better explain its silence. Upon information and belief, OpenAI was seeking to avoid implementation of a
    hard line rule to refer planning of real-world violence to authorities, perhaps due to how frequently its
    product is implicated in threats to human life. Requiring such disclosures would be incompatible with the
    company’s public position that ChatGPT is safe. It could also threaten the valuation underlying OpenAI’s
    anticipated initial public offering. Rather than expose those risks, OpenAI accepted the consequences of
    its silence. A mass murder in the only secondary school in Tumbler Ridge followed.

    Accordingly, the Crown, led by Attorney General Sharma, and SD59 jointly bring this
    action to hold OpenAI and Sam Altman accountable for designing a dangerous product, distributing it to
    every home with internet access, ignoring the warnings of their own safety team,
    refusing to notify authorities when they knew the shooter was planning gun violence, inviting the shooter back onto the
    platform after deactivating the shooter’s account, and choosing corporate self-interest over the lives of
    children. They seek compensation for the not just foreseeable but known harm OpenAI inflicted, the
    damages that they incurred and are incurring, and injunctive relief to ensure that this tragedy does not
    happen again.

[1] https://cdn.arstechnica.net/wp-content/uploads/2026/09/Briti...

In that case was the shooter using the LLM to plan the attack?

That could be a meaningful difference

That's what the suit alleges. Also that it had been automatically flagged to an internal human review team who had concluded that there was a credible, specific risk of harm to others and recommended notifying police, and that it was OpenAI leadership that rejected that recommendation.

You can see it all in GP's link. It's very readable.

... Wow. Well that's not at all what I thought happened.

I figured it just didn't get flagged properly.

So mentioning killing in passing is fine, but planning with the tool is not? How should LLM providers draw the line? I'd prefer if they didn't scrutinize any of my inputs to see if I am committing a felony or not in the first place - but that ship has sailed.

If it’s just a journal, how is it different then Google Docs?

...yes?

They draw the line using a team of humans who evaluate whether the threat is real and give a recommendation to management.

If you operate a gun store and someone reveals to you that they intend to use your product to shoot up a school you have a similar moral obligation, I think. This isn't a special case.

But it is, it somehow only applies to LLMs. What about doc apps? Todo lists? Chat apps?

There are tons of hosted app services where you can write private expressions. Why are only LLMs subject to monitoring?

You're really stretching here for some reason I can't fathom, but yes - if you write a credible threat to shoot up your school on a crumpled up paper and someone finds it, they should report it. I'm not sure that complicating this with which kind of app it's written in is all that useful ...

There's a big, big difference between "finding by happenstance" and "automated monitoring and analysis of everything written with invisible alerting of authorities". It's a step towards a digital panopticon and there are more than a few objections people would have to living under a digital panopticon.

Uh, no, text you write in private should not be automatically turned over to the police.

Sure; but writing to an LLM is not "in private". That's the point of the original post.

The average consumer thinks it is.

I don’t see how writing something you clearly intend to be private can be a threat.

How many mass shooters left diaries, letters, memoirs containing threats? I don't see why you don't see.

Innocent until proven guilty. Every citizen has rights until there is reasonable suspicion, probable cause, etc that they are commiting a crime.

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated...

The court is going to have a look at it.

If you want to be secure in your papers, maybe don't type them into a giant data collecting machine.

How many of them drank water and/or respired oxygen?

The false positive rate of the test that you suggest is frankly atrocious.

Is this a point you're arguing seriously? If you find someone's plans to, for example, shoot up a school crumpled in a note on the ground that should be...ignored? Or what? I struggle to comprehend what you're actually arguing for here.

I think I pretty much agree with you that if someone finds evidence of danger to someone else's life then it is a fair expectation that they do something with that evidence. And despite that I still believe that is not a good enough reason to eliminate someone's right to privacy, and yet still that right doesn't mean that we should be able to demand privacy in all products. I think it is fairly clear that these big LLMs are not private products and we shouldn't expect them to be, but I'm still in support of offering private alternatives that people could use without having every input scrutinized.

I guess I just see a lot of "X, but with an LLM" getting different treatment than just X recently, and it's confusing.

I don't think that most (any?) of these other cloud services monitor for this stuff, or that government thinks they have an obligation to do so.

I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.

For technical people, this is incredibly old news.

For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.

One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.

"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.

You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.

Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.

I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.

And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.

Those in power have an interest in making criminals seem dumber than they are. Those who get away with crimes aren't part of the calculus either.

Those in power are often dumb criminals themselves. The fact that the system will break its back for an excuse to not prosecute them doesn’t make them skilled operators.

> Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway.

...but the last sentence if kind of important, right?

We only know about the criminals who got caught. Which probably means they did something dumb.

It's conceivable to me that there could be many—maybe even a majority—of criminals we don't know about because no one ever caught them. Maybe they committed fraud once, decided not to push their luck, and lived the rest of their life quietly. Maybe they killed someone and made it look like an accident, and no one ever suspected anything. How would we know?

You can look up crime clearance rates. It's a little depressing.

Here's an FBI chart for 2019 [1]

Though some countries like Japan and Germany have really good clearance rates for homicide. [2]

But more heartening, it seems like it doesn't really matter how many people the cops catch. Crime rates are still generally on a long downward trend (with some bumps along the way). [3]

[1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...

[2] https://en.wikipedia.org/wiki/Crime_clearance_rate

[3] https://ourworldindata.org/grapher/types-violent-crime-rate-...

And even those are only the crimes we know about! I imagine a really smart criminal wouldn't let anyone know a crime had taken place.

Zodiac killer was never caught and it seems he already died a few years back - it was established some time ago through some teen's DNA whose grandfather was ex-cop or something. Don't quote me on that but it's on the net somewhere and was there before AI slop happened.

Basically a lot more evil goes unpunished than most realize, because carma is a thing only when it's a thing - sometimes it just isn't.

I actually completely agree with that. I guess it depends how you want to slice that in terms of categorization.

Which is to say: If a criminal gets away with a crime and faces no punishment, not even public scorn, are they really a criminal? Or did they just work a situation, even of their making, to their advantage?

I don't particularly know where we should draw that line. But yes, my statement does deserve a big caveat there.

> getting by honestly is just a way more comfortable

I think it's also much about self-esteem -- am I proud of myself? And criminals probably think highly of themselves when they succeed in their crimes.

It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.

TV is a human in the box. A phone is internet in the box. It's all implicitly local, because you clutch this internet in the box right in your hand. Synchronization sends data directly from one internet in the box to another internet in the box.

People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.

Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.

Or 'public/private information sharing' partnership ( hello Canadia ^^/ )

I actually hope someone is reading it. It's so sad thinking I go to all this work to be clever and no one reads it.

Of course, if I don't want anyone reading it, I sure as heck don't put it online.

People not reading it is unfortunately pretty close to the truth.

You can bet that everything sent across the Internet is stored somewhere. But read? Yeah right. People don't seems to realize that there are 6B+ people online and just how big a number 6B is. If you have a staff of 30-40K people (like the FBI or NSA) and they spend 40 hours/week doing nothing but reading Internet posts and the average person spends 4 hours/week writing Internet posts (likely a huge underestimate) and the FBI agent can read 5x as fast as the person can post, then the collective manual surveillance capacity is about 2M people. That gives a 1 in 3000 chance that a random Internet user would be surveilled (though it is decidedly not random). With more realistic numbers of maybe 5000 analysts reviewing potential threats, spending 10 hours/week on it (so they have time to actually follow up on threats, attend meetings, communicate with their boss & coworkers, etc.), and the average person spending more like 25 hours/week posting on the Internet, that's a surveillance capacity of 10K people and your chances are more like 1 in 600K.

This was a reasonable argument until about 3 years ago.

I assume you're referring to AI?

AI will give you only what you prompt it for, and the prompts are still designed by human analysts. What's changed with it is that now the 10K people surveilled will be anyone that "fits" in a category that the government deems criminal (say being transgender, or foreign, or left-wing, or criticizing ICE) instead of actual criminals. It shifts targeting, not capacity. If you wanted to be an actual criminal and are willing to do it in creative ways that bear no resemblance to other major categories of criminality (or lets be real, political disfavor), there is no time better than the present.

Sorry mate. The one thing llms effectively guaranteed is that short of wonks like me, no one is actually reading the source. Everything is a summary of summary of a summary.

FWIW, I feel you; I too try to leave internet a mildly more amusing place to be.

reading what exactly?

For some, online discussion is just decentralized journaling

You just read some of it.

> I'm more confident that local apps on my Linux desktop are secure

Why should you be? Linux gives us software we can reasonably say we own, but not hardware. Everything you type into your local linux apps can be being monitored by your processor and whatever is sitting in the CSME/PSP subsystem which you don't have permission to access, but which is always running even when your computer is off. We need fully open, documented, auditable hardware if allowing any third party access to our devices means our private documents will make us all into criminal suspects.

Most of the hardware isn't made in the US, so either the spooks would need agreements with every offshore business, intercept every hardware shipment, or convince every vendor operating in the US, even if they are not locally owned, to do this for all hardware sold here, and convince all of these offshore vendors not to ever leak the fact that it is happening.

On top of that, the data has to pass through several layers of routers and access points we control, which makes at least the fact of exfiltration visible. Plenty of people have a very strong vested interest in preventing data exfiltration from company hardware and so the government would have to somehow get in touch with every one of them to tell them to stay quiet before they leak any hints of it happening.

It's possible for targeted shipments, but the prospect of it being done for all consumer hardware sold in the US is a bit far fetched.

> Most of the hardware isn't made in the US

We know for a fact the government does intercept hardware shipments already on some scale. It's a lot easier when there are only a few chip makers being used for most devices. Intel and AMD alone cover the vast majority of the CPU market for desktops and laptops. It's basically the same situation for the wireless chipsets in every mobile device. No need to worry about dealing with about every cell phone manufacturer directly when you only need to hit up the extremely small number of companies every manufacturer has to get their chips from.

> Plenty of people have a very strong vested interest in preventing data exfiltration from company hardware and so the government would have to somehow get in touch with every one of them to tell them to stay quiet before they leak any hints of it happening.

Whistleblowers are extraordinarily rare. Edward Snowden worked with many many others. Any of them could have come forward at any time, but they didn't. Only one AT&T employee came forward to tell the public about Room 641A (https://en.wikipedia.org/wiki/Room_641A). The government marched into AT&T's building, took over part of their offices, rerouted their network and the whole AT&T backbone into their offices and through hardware. You can bet that more than one person noticed that. Government has no problems at all with going to a business with guns and gag orders and telling people to keep their mouths shut. Companies have no problems keeping quiet about what's happening either.

No one who isn't directly in the know is going to notice if someone's CPU (which is running it's own network stack) sends a few extra encrypted packets going to the servers of major internet companies (microsoft, cloudflare, apple, google) to either be collected at those end points or just picked up as it passes across the internet backbone.

Maybe if we had a ton more competition in things like chip makers, ISPs, operating systems, etc. it might be more difficult, but the way things are it'd be easy.

Yes, I mentioned the Tailored Access Operations NSA unit in a sibling comment. The thing is, open source hardware won't save you from any of this unless you have the manufacturing skills and equipment to build it yourself. Just like with open source software, if you aren't building it yourself, with your own hands, you can't be sure that what end result you have actually matches the code that you've read, unless there's a published checksum that you've also verified is real. Quite a bit harder to do with hardware than software though, especially since every single piece of the stack, including monitor cables, have been targeted by TAO.

Companies like Google and Microsoft pay top dollar to build special workstations to interact with production systems. These are not just special because of the software they run, but also the chips these machines are built with.

Sure, I'll take your comment at face value, not all consumer hardware has spyware, but it cannot be ruled out now, in the future, or at least now for targeted high value customers.

That’s… really not true or at least not as dramatic as that statement makes it sound.

Those “special workstations” are mostly about lack of features they view as potential attack vectors (external or internal motherboard ports, replaceable/non-soldered components, thunderbolt, anything extensible or “repairable”) and mandate certain hardware features (hardware tpm, locked secure boot, locked UEFI). Other than that, those workstations are Lenovo, Samsung, or Dell machines running typical chips. They may pick certain CPU models or chips but that’s mostly about avoiding new or “cool” features that may not have matured enough yet. Most of them are moving to thin clients though. Where you use that “special workstation” to remote into an VM that can access production systems. And that part is partially about controlling the software running in the VM and making sure updates can be forced “offline” on the VM even if you haven’t connected to it in few weeks.

> but it cannot be ruled out now, in the future, or at least now for targeted high value customers.

Well, sure, there was NSA's Tailored Access Operations unit which interfered with all kinds of hardware, even computer monitor cables:

https://www.nbcnews.com/tech/tech-news/report-nsa-intercepts...

For these kinds of operations, open source hardware won't save you unless you build it yourself, much like open source software. At the end of the day, you're hoping that the pre built thing matches what it claims to be.

The hardware case of TAO is still targeted and didn't rely on the manufacturer's compliance and secrecy for their entire line of products, though. The scale of that to my mind is quite the qualitative difference.

This is (seems) impossible.

You can have open and auditable hardware, but how can you be sure that the hardware you buy is exactly the same hardware as what's in the Github repo?

It's true that you'd have a hard time knowing a chip wasn't swapped someone on its way to you. There are a few things that would help but nothing easy. Getting the hardware into retail locations so that anyone could walk in off the street and buy a system with cash. Random spot checks, chip by chip. Packing the system in colored beads/rice to prevent tampering after it's in a box.

At this point, even if I could never be sure my system was secure, just knowing that systems were purchased, tested, and verified would make me feel a lot better. It'd be nice to have confirmation that it's even possible to make and sell a system that isn't backdoored in this country.

If we completely give up on assuming privacy, we lose even more privacy, because so much is based on the expectations of a "reasonable person".

Our devices, even Anthropic in this case, set an expectation that doesn't match reality. The software and devices we use are marketed and sold to us to be used for our private documents, photos, and conversations. I can't blame people for using them for their intended purpose. They shouldn't assume that someone is carefully watching everything they do and judging their every word and action looking for evidence of crimes. If we want people to think of the devices and services they bought and use as belonging to someone else and without any expectation for privacy we should really start with forcing companies to change how they market their products and services to reflect that reality.

> They shouldn't assume that someone is carefully watching everything they do and judging their every word and action looking for evidence of crimes.

This is silly. Nevermind the government; you should always assume some Redditor will pore through OSINT data and leaks looking for anything they can use to dox you and wreck your personal and/or professional life.

> apps on my Linux desktop are secure

That's definitely not the good word to use. It is most likely made with the intent to be privacy-friendly, but they are unfortunately anything but secure (including the whole Linux userspace), and especially in the age of agents it would be the best if everyone understood it that you are a single bash/npm install/malicious PDF away from everything bad happening with your data. But rest assured, your video driver won't get updated!

https://xkcd.com/1200/

> I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications.

I'm also puzzled by the phenomenon of using ones real name on the Internet (outside a professional context). I believe this began to occur around the time facebook became popular.

I've always used my real name, since I first got on to the internet in the early 90s. It's never been a problem. Big Tech and your own government know who you really are anyway, as do the big online advertisers. Pseudonyms are no defence against that. There is a potential risk from griefers, but they have ways and means as well.

I wouldn't be surprised if a dedicated enough IT at a company is not monitoring your enrolled devices microphones. Just listening in the background for shit-talking the company.

It is funny that you mention that, because at home I do/hear/watch all sorts of things, so I suspect transcripts look fun and may make management feel a little icky ( makes me wonder ). Honestly, one would think it is a reasonable position that they would not want to know too much:D

I understood this a while back when seeing that hedge funds etc require employees to lock away their devices.

no outside device comes in - no inside device goes out.

You know, at my previous job the company had a semi-strict policies with non-enrolled devices. If we didn't enroll our devices we were locked out of some company systems, but we could still use email and teams (which is like 99% of my use-cases for my phone).

So I obviously opted to not enroll my phone, but that came to bite me in the ass one day when I needed to get a train ticket from my company email (it was a business trip) and the policy prevented non-enrolled devices from attaching or downloading attachments...

Felt like a fool grabbing my laptop to let the train attendant scan the QR code.

Worth it still. Next time print the ticket on dead trees?

funnily enough, afterwards I would just forward it to my personal email. IT never bothered me about sending .pdf attachments from my company email to my personal email...

I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications

Wow, so we're assuming police state plus it's your fault if you didn't know...

and this assumption is wrong how exactly when viewed through the lens of real-world events?

I'm not talking about the lens of morality nor even the law, because it's obvious this shouldn't be the case but in actuality, it is - everyday more true than it was yesterday.

I don’t think Google is reading all of your email in this way, or your phone text messages being read in this way, or say iCloud or OneDrive files being monitored in this way.

Certainly it shows that agents like Muse are a complete nonstarter for anyone doing anything that needs to be private. Even if you never talk about it with the bot, merely having sensitive information on your hard drive means it can be sent to outside servers where, if it hits some safety filter, some probably low paid employee is going to read it. Say you’re working for a public company and have files on your computer that could be material nonpublic information, now potentially some content monitor is going to be seeing that and trading based on it. It’s completely untenable to have everything you put on your computer be subject to human review at some tech company.

All your communications which are available are being swept up by efforts like prism and stored.

I think you can count on governments accessing that information where it suits them, and you can definitely count on unscrupulous people like Altman within tech companies to do so too.

I’m not sure what the answer is here, but it’s naive to think that nobody is accessing the information you give away by sending it to cloud AI providers.

Everything you're saying here is correct. It operates on the power of naivety.

Anytime the strong encryption debate comes up it's actually about maintaining the effectiveness of unencrypted surveillance in the AI era.

This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.

Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.

I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.

Both things can be true.

They want stable rules they can follow, which will limit their liability as long as they stay within them.

They also would like those rules to be as restrictive as possible towards their competitors.

They've spent close to $2T so far. At that scale the legal issues they've had are just the cost of doing business.

I'm quite sure that if there wasn't the existential threat of a lack of a moat, they would not be pushing for regulations at all.

It doesn't have to be an existential threat. As long as the lobbying is cheaper than the possible legal costs, it will look reasonable to lobby for rules.

The legal issues can sink their entire business model, not quite the cost of doing business.

A lot of things could sink their business model including opening the can of worms of the wrong regulations.

After all the claims of "the product we are developing could cause the extinction of humanity" I'm honestly surprised Florida is the only place so far to seek an injunction preventing further development of the product.

It's really hard to take these companies seriously or at face value about anything they say.

As restrictive as possible towards their competitors, while being as permissive as possible for them. Getting control of the regulation making process means they can make sure they don't end up too constrained anywhere that would hurt their bottom line.

That's the conspiracy theory, but there's no good evidence for it, and it doesn't really make sense in the long-run.

It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.

You're assuming the open models will follow regulations though.

No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.

The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.

No good evidence other than their openly stated desire for that regulation? Dario and Sam have both made open pleas for it, repeatedly.

Liability is enough of a reason

And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.

As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.

How much do you love Big Brother?

Wasn't this because the photo was made "public" as in, shared through Google Photos?

If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.

Most people don't realize that it is 99% like posting it on Facebook.

It isn't like posting it on Facebook. It is like emailing it, because most people don't have the link.

If it was a password entered into a form would you still consider it protected by obscurity? Both are the same level of security.

The URL may be logged by the ISP, university/company or whoever is providing the connection, antivirus, windows recall, browser history... But IIRC passwords are send using cryptography if the site uses https.

The URL is sent using http inside the https connection, so it's encrypted too. Your ISP shouldn't be able to see it unless they have root certs on your device.

It's not impossible that someone sees the URL, and if it's stored in an email the backups or local clients could store it plaintext, but I don't see how the URL is any more likely than the password to be sniffed during the connection itself

Its nothing like posting it on Facebook. Google doesn't have a recommendation engine that tries to share the image with random people, or a search that image will appear in, or a set of subscribed people who will be notified.

Its a random link someone would have to guess.

I can't read past what the paywall allows, but here's the nyt article: https://www.nytimes.com/2022/08/21/technology/google-surveil...

> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech

Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.

Or something like leaking every non-corporate ChatGPT user's chat logs (including temporary chats) to the NY Times.

Or making up stuff

Don't know why this got downvoted? This is a problem with epistemics.

It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.

Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).

> welcome to negotiate this in enterprise contracts

I really hope you mean that in an insulting way to the industry and current legal situation, not as an actual solution.

Where's the insult? Lots of customers have such agreements.

That anyone smaller than an enterprise gets screwed. It's like responding to a problem with "let your butler handle it".

Individuals can sign up for enterprise agreements. An enterprise can consist of one person. Or run a local model.

It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.

> Individuals can sign up for enterprise agreements. An enterprise can consist of one person.

Custom negotiation is a big difficulty for individuals and the pricing will probably have fixed overhead that becomes pretty ridiculous when it's not spread over a hundred or thousand users.

> Or run a local model.

Much better advice. But still an obnoxious amount of effort and restriction to avoid spying.

> It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.

I want it to have the same privacy as using a local program. Does it still sound ludicrous if I put it that way? I'm not talking to a person that would have to suppress their own conversations later, and there's no need for any data from this chat to escape into the outside world. It's like wanting a phone line where the phone company won't listen in; even if doing that is standard procedure and I can walk away it's still gross.

In the case that the company running this program for me is served a warrant, I accept that saved data would be shared. But proactively screening my data for a reason that isn't to help me is bad. We should fight to make that easier to avoid.

Also you're talking out both sides of your mouth when you say it's "ludicrous and unrealistic" right after telling me I can negotiate that exact feature in an enterprise contract.

Yes, it still sounds ludicrous if you put it that way. If you want a contractual agreement for privacy then pay for it. Everything is available — for a price. But a civil agreement can't bind a counterparty to conceal evidence of a crime.

Privacy should be an offer from the start if it's not mandatory. It shouldn't need special negotiation.

> But a civil agreement can't bind a counterparty to conceal evidence of a crime.

Are you talking about a different scenario? In a chatbot situation, the crime of "threat" requires snooping to even be possible. If the data is private there is no crime to begin with.

Wow. Now that is a weird and interesting take. Paying for privacy. Wow. I guess next we will be paying for respect too.

this thread is about ordinary people using chatbots for personal use, not enterprises for work

I have expanded the scope of the thread to make it more sensible.

The more sensible thing would be to link to any example of any lab anywhere offering enterprise solutions—with the opportunity to negotiate—to individuals.

People are confused at your take because it’s based on a reality that so far you’ve only described as the better/obvious option for individuals. Companies treat corp customers and individual customers differently. This isn’t new.

My company did negotiate the data policy amongst other things for our enterprise plan, I was involved in the process. I can say with certainty that they’re not picking up the phone for _one dude_ who wants the same thing.

I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.

Perhaps but no indication of that so far. Agents are happy to set up *arr stacks today.

They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.

Sure, and they could be a front for space aliens choosing who to abduct. Anything is possible!

You mean forward to the copyright holder or more likely a copyright troll who paid the copyright holder for the right to represent him as lawyer.

We have weaker versions of this already.

I had to move some games from C:\Games\RandomWarezGroup\GameName to my Steam directory, because Claude refused to reverse engineer an asset format due to piracy being incredibly immoral and illegal and blah blah.

It’s trivial to bypass this by saying something like you’re the games developer trying to track down cracking groups

But it sure is annoying

It's a minor annoyance for now, you can pretty easily convince an LLM of almost anything with a plausible-sounding reason.

The issue is when Anthropic flips a switch and instead of telling me "no" it calls the report-super-dangerous-crime tool.

I'll be glad for open models when the day (inevitably) comes that the proprietary LLMs no longer work in my interests.

> LLMs no longer work in my interests

As articles like these show, cloud-based LLMs don't work in your interest today.

Is it in someone's best interest to let them die in a shootout?

Level 1:

Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?

If so, or if you can imagine a journal or always-listening assistant hearing your muttering thought. Should you be charged with a crime for that? Who is the victim and what are the damages?

Level 2, the slippery slope:

Suppose you travel to a country where your sexual orientation is illegal. If your AI company or social media provider knows your orientation, should they be compelled to reveal your crime? Why or why not?

> Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?

This - specifically with a shooting, is basically the plot of The Drama (film) from earlier this year.

No AI in the movie plot though.

That it's a tough line to define doesn't mean there isn't a line, unfortunately.

For them to die in the shootout, you have to assume there was actual intent to execute said shooting.

"According to the arrest report, a user identified as Carli made a statement on Sept. 26 saying she was going to 'shoot up' the Lee County Sheriff’s Office. Investigators say the same user made another statement the following day saying she had gotten a new gun."

At a certain point, intervention seems appropriate. This meets my personal threshold, and I'd hope a friend would do something if they came across that in my journal.

Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.

There’s no recognized inalienable right to keep and bear models/GPUs.

That day was yesterday. LLMs from big tech regularly refuse to do what you want.

I heard an ad on the radio this morning from the local Internet support/WordPress/website developer/seo company that they've moved from seo into astroturfing for _LLM_ input. They advertised that they'll make sure you're company is the one that gets returned and trusted when other companies search the web via agent. If this works for agentic search, it probably works for _training_ as well, so good luck finding an open LLM that's been trained to follow your interests. Has there been any research into whether/how much LLMs are more likely to recommend companies whose names they saw in training over similar but smaller or newer command also returned in search results? Given how well advertising works on people, I can't imagine that it _doesn't_ work on LLMs too.

Ask it how to evade astroturfing.

Wouldn't they have to fine themselves first?

They were fined for their illegal downloading. More than a credit card limit.

And dramatically lower than their expected value from the copyrighted material they scraped

Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude

Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).

> in private chats.

Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.

> they're chatting with Big Tech

They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.

> people need to get it in their heads that they're not chatting with their secret BFF

I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.

People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.

Problem is that even not using those apps, the apps you currently use might have turned more hostile.

It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.

How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?

The only way is for them to suffer consequences of mass surveillance.

But then we have to as well ={

A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.

A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.

You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.

The Silent Generation, version 2.

They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.

I think this is the only business model that makes sense, monetarily, for current LLM CapEx and OpEx. The value is not in what it can do for consumers or how much money it makes, but how it can shape consumers through knowledge control. Think of how much a totalitarian government would pay to keep citizens from thinking certain things.

I don't even think government would be the primary customer. Advertisement is already that.

I'm hoping and actually expecting that eventually there will be a simple way to locally host a small model that is good enough for simple things. For a lot of use cases, the capability of free chatgpt one year ago was more than enough.

Maybe there already is! But last I checked it was a little bit of a mess of manually installing things from multiple websites with little documentation.

I'd expect someone's BFF to call the police if their friend is making credible talk about shooting up a school or whatever. I sometimes feel like privacy talk online can be seriously divorced from how human relationships actually work.

Sure but you wouldn’t expect your filling cabinet to call the cops on you. Because your filling cabinet doesn’t read your papers. The problem is the spying not the acting on credible threats of violence.

> Now, everything you say or write can be automatically screened for red flags

Considering that many people will share their deepest thought with LLM, it might start looking more and more like attempts of precog agents from Minority Report...

> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech

It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment

This is something of a HN trope, "I'm sympathetic to X 'cause they tried to do their thing one way and people complained so now it's OK if they do it the opposite way, even if that's kind of F'd up too".

The problem with that is, "no, they don't have to do their thing. They have no given right to do it. If they do it, they should, they have an ethical duty to, do it right..."

Right, the difficulty is partly that they can get a negative headline from any choice of behavior.

"Anthropic failed to report murderer's threats to authorities"

(or "Chatbot knew man was planning murder, yet company did nothing")

"Anthropic reported private chats to authorities"

(or "Arrested for chatbot fantasy")

To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".

Not sure why so much sympathy for any company that makes its bed and now has to lie in it.

It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.

Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.

As if it is just nannyism marketing by tech companies. The EU tries to bring a new nanny law into legislation every other month

Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.

No sympathy for whom?

For Anthropic? They don't need it.

For the woman, whose entire crime was using a chatbot that runs on Anthropic's server, and wouldn't be charged or guilty of any crime if she used a similar chatbot on her own machine to write the same exact thing?

The woman whose only crime was using the cloud to store private information (diary entry, stream of consciousness, thought experiment, etc) assuming it will stay private, as is normally the case?

The woman who was jailed on a technicality that allows the government to treat the 1:1 conversation with a robot as a public threat because the fine print says that the corporate has the right to snoop on the conversation?

I hope the the woman gets some sympathy from whoever reads this news, because it's a travesty and perversion of justice that this took place and Anthropic assisted in.

This crap makes KGB look good.

We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.

I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.

There'd also be fewer people falling for apocalyptic Rationalist delusions.

Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.

You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?

My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).

There'd also be fewer people falling for apocalyptic Rationalist delusions

Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.

Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.

> There'd also be fewer people falling for apocalyptic Rationalist delusions.

Um, this is coming from the researchers building these models. So, good luck?

>We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed

Not applicable.

If the woman was chatting to a local LLM, there would be no way to charge her for a goddamn thought crime.

Which is what the government did here, with Anthropic's assistance.

Thinking about committing a crime isn't a crime.

As for treating the "threat" as "public".. something tells me if the court ordered to share something publicly, telling it to a chatbot would not count.

>Also: people need to understand "not your computer, not your data"

Rrrright, because it's so easy for people to know when stuff from their computer is transmitted to not their computer.

There's nothing physically stopping those companies from scanning data on your computer either.

And legally, surely the state that treats convo with a chatbot as public when it suits them would prosecute mega corps for overly thorough telemetry when it also suits them, right?

Why people here accept this as normal is beyond me.

>I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.

There are people with education here that don't fare much better, so I don't know.

> they're not chatting with their secret BFF

If a person told their real BFF they were planning a shooting and went through with it, couldn’t the BFF be charged with accessory?

I don’t think so under historical English common law. If you lift a finger to help, i think so. If you see everything with no aid whatsoever, I don’t think so

Curiously predates US law, it’s old English common law still operating as a principle

IANAL blahblah

> Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs.

They did and they do

https://www.bbc.co.uk/news/technology-44699263

I never considered online services safe. What I object to is active scanning of content on devices. This should not be allowed.

Yeah. Damned if you do, damned if you don‘t.

> I have some sympathy for Anthropic here [...] So from their perspective, it's damned-if-you-don't, damned-if-you-do.

On the other hand, they did put themselves into this position deliberately.

By offering a product?

Show me any product, no matter how simple, that has no safety vs utility tradeoff.

They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.

For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.

that tells users upfront "we can read whatever you type in here"

(and not buried on paragraph 56 of a 20 page TOS that no one reads or understands)

You would first have to demonstrate the utility of whatever product Anthropic is offering.

>By offering a product?

Yes.

>Show me any product, no matter how simple, that has no safety vs utility tradeoff.

Show me a product that was not offered willingly.

Observe that Claude isn't one.

it was in the article

This isn't normal, this isn't how technology has worked.

Google doesn't report people to the police for the private (non-CSAM) contents of searches or emails.

You could definitely go through people's shit before, but Big Tech generally had serious prohibitions and a stronger presumption of privacy about this sort of thing.

CSAM was an exception where they theoretically had the evidence of the actual crime, rather than writings alluding to them. And even that has been problematic.

Frankly, the charge here is total bullshit and should be thrown out. The law is meant to prevent people sending threats to others, not keeping notes. If this charge sticks, the law should be changed.

Reply on news.ycombinator.com