Microsoft is spying on users of its AI tools
schneier.com
schneier.com
> OpenAI may securely retain API inputs and outputs for up to 30 days to provide the services and to identify abuse. After 30 days, API inputs and outputs are removed from our systems, unless we are legally required to retain them. You can also request zero data retention (ZDR) for eligible endpoints if you have a qualifying use-case. For details on data handling, visit our Platform Docs page.
Compare with the Google Docs privacy policy, for example:
https://support.google.com/docs/answer/10381817?hl=en
> Google respects your privacy. We access your private content only when we have your permission or are required to by law.
I think it is reasonable to expect the same from LLM API providers. The fact that they all currently do mass surveillance on users is bad.
That can be very vague. does TOS gave them the permission? Does law can eb interpreted to require them to check for CSAM and pirated material? What about access by Iran or North Korea agents? If you are pushing things to cloud, these questions will be asked.
I’ve managed a very large tenant with a number of carve out T&C. I’m aware of at least a dozen times that settings reverted during upgrades or migrations. O365 is a huge service with a lot of humans and duct tape holding things together.
Unless you’re auditing them, you have no idea.
I don't think these TOS are out of this world.
Next up, "Major television broadcasters spy on soccer players during Champions League games".
The alarmist headline is warranted, and the fact that it's common doesn't make it acceptable. I welcome more awareness of this kind of lopsided ToS that enables such user hostile actions.
You don't have a 'right' to use the services. So the answer to your question is "It depends if the service lets you, so 9/10 cases, no. Rightly so"
Edit : I can see a lot of people don't like this. But like it or not, right or wrong, that is how it is. Expecting anything else is just fanciful.
I don't have a Windows machine and didn't check, but I'm guessing it's a lot closer to the second than the first.
In principle I don't think anything is "wrong" with keeping a record of AI chat sessions, but you do need to know about it so you can modify your behaviour as you see fit (i.e. ask different questions). If there's a camera pointed at you then some might decide that the nose is best left unpicked and those balls will have to remain unscratched.
This is really the problem with a lot of these things: not that they're doing these things as such, but that it's so hidden and obfuscated that you need to spend an hour or more trying to understand it, and even then you only know what they "may" do, not what they "actually" do – a lot of these privacy policies are so vague and full of qualified language that they "may" do almost anything.
And even in recording conversations there's nuance. Is this recorded with your full account detail, or is that information stripped? Are all conversations recorded or only some? Etc. etc.
That is often a crucial issue in these discussions around big tech, enshittification, and so on.
But "simply don't use Windows" is already a lot more tricky, because so much of the world runs on it. You and I can get away with just running Linux, but we're not normal people in this regard.
By analogy, consider mental arithmetic vs. electronic calculators and spreadsheets.
Being able to do mental arithmetic is an important skill, arguably the only way to have a profound and deep understanding of many things. At the same time, calculators and spreadsheets allow an individual to do things that just would be totally out of reach without these tools.
Some people argue that generative AI has already reached this point, and I think that's plausible in a few narrow domains. It's also plausible that the domains to which it applies keep broadening.
sounds nice
Informed consent to me (take the example of a phone or many apps), based on the level I have to break it down to people for them to understand, is something like "we will see you whacking it to hentai, we will know when you take a dump, we have enough of your data to derive private information you never gave us, we have all the data necessary to totally ruin your life socially and economically, and it will all be leaked to criminals that will profit from your demise, dont worry though, we will give you a 1 year sub to mcafee antivirus as a consolation prize."
I think we can at least agree that ^^that happens to people. So, whos going to be the first company to not veat around the bush and just admit that reality? That, is why I say informed consent sounds nice, every company that can keep a skilled contract lawyer on payroll has an incentive to downplay their spying. I dont know in what version of reality informed consent could exist, because the incentive is fairly obvious.
He'a referring to an idea that sounds nice in theory and doesn't work in practice.
It's a common English phrase.
And the phrase works well, the concept of informed concept sounds good in theory and never works in practice.
Anyone who has observed tech companies for decades knows they don't ask for your consent before they make any consequential changes. Tech companies do whatever they want until they get sued or regulated. It's always been this way.
https://knowyourmeme.com/memes/we-should-improve-society-som...
I strongly agree with you that this story is a nothingburger.
But as I note your reaction, I take a different interpretation. The implication that statements like "Microsoft spies on X" are truisms, is the news. We obviously all now accept as a given that these are tautologies.
In other words "Spies spy on X" -> (is to say) "Microsoft = spies", not just in this sentence/context, but in all contexts.
Terms, conditions, legality, illegality, hostility or friendliness all recede into irrelevance hereafter.
"Your conversations are processed by human reviewers to improve the technologies powering Gemini Apps. Don’t enter anything you wouldn’t want reviewed or used."
"How conversations improve Gemini
Just by having a conversation with Gemini, you’re making Google services better, including the machine-learning models that power Gemini.
As part of that improvement, trained reviewers need to process your conversations.
So when using Gemini, don’t enter anything you wouldn’t want a reviewer to see or Google to use.
Your Google Workspace content, like from Gmail or Drive, is not reviewed or used to improve Gemini.
You can turn Gemini Apps Activity off If you don’t want future conversations reviewed or used to improve machine-learning models, turn off Gemini Apps Activity"
It is only kinda 1/2 clear here.
If you use a modern MS product, you are sending them lots of data, full stop.
Many of the privacy apps and services I use explicitly set out to collect as little information as possible. It's sad that people have been conditioned to think spying is appropriate or that it has to be this way for the internet to "work".
Not sure how this is a surprise.
Compare that to, say, Nvidia, who sends every single titlebar value, as well as every single click you make to their servers (this data sends even if you opt out, as it is classified as “necessary”). Probably the most invasive spyware you have installed today.
Certainly these firms are bound by contract, but the stuff I've seen people ask these chatbots – health information, sexual information, highly personal stuff you might only share with your shrink... well, it's being read by the 22 year old marketing intern as well. Unencrypted, in the clear, copy & paste'able, and attributable to your username/ID
That's not to excuse it, however. I think it's already time they all implemented a privacy stance where they are only allowed to view chat conversations if the user gives them explicit and revokable permission.
AI is built on data. The more you spy, the more data you get to train AI, the more you train AI, the better/more valuable it gets and the more people want to use it to give you data. It's a cycle.
Everyone of these platforms is scraping your inputs to train their next model.
How is it not obvious they would also detect abuse?
If you're sending the data to someone else's computer and it's not encrypted, you're.... sending the data to someone else's computer.
Do people not understand this?
They apparently have no idea.
So yeah, I think that people legit don't understand. The gap between tech literate and hacker is problematically large.
It'll either end in a dark cyberpunk total surveillance and mass manipulation state or a kind of Butlerian Jihad where people reject networked cybernetic systems en masse. Maybe the first happens followed by the second.
I'd rather retire of unremarkable means in a functioning society than be wealthy in a dystopia of my own making.
Intelligent commentary will need to differentiate between public cooperation layers, non-public cooperation layers, and non-public adversarial layers .. because, they are all active.. all of those.
https://www.whitehouse.gov/briefing-room/statements-releases...
I suppose other people might ask more private questions, but I have trouble coming up with examples of private things that I’d want to ask an AI chatbot about. And there are plenty of things to ask about that aren’t personal.
This is true of everyone. Do you need privacy all the time? Presumably you're okay with your Hacker News comments being public.
Like, are we letting ourselves be herded towards a future where it's normalized that all search must be done by authenticated users, and "classic search" is effectively deprecated ten years from now?
>OpenAI are “spying” on ChatGPT sessions the same way that email providers “spy” on your email to filter spam. This sounds like a variation on “Google is reading your email”.
Yeah, but Google is actually reading your emails. They build a profile of everything you buy from emailed receipts, and use that data to target ads at you. It's definitely not about improving the user experience, like a Spam filter accomplishes...
For what it's worth, the US military is also evaluating LLMs in the same way the Chinese and Russian hackers are. I imagine - or at least hope - they aren't using OpenAI's hosted ChatGPT and are getting copies of models they can run on SIPRNET or JWICS.
Not sure why anyone would think that terms that allow surveillance are not functionally equivalent to evidence of surveillance.
But it seems like this thinking is what lets people trust the aptly-named "privacy policies" used by so-called "tech" companies to make users more comfortable with surveillance.
If a so-called "tech" company makes statements in a "privacy policy", something like, "We do not do X", and there is no way to verify whether they do X or not,^1 what purpose does the privacy policy serve. If in fact someone acquires evidence that the company is doing X, it is already too late. X was not prevented.
Whereas, if a so-called "tech" company includes provisions in its terms of service that allow it to do X, then what functional difference does it make whether the company does X or not. They could do X at any time. Anyone who agrees to the terms has given them permission to do X.
In general, companies do not ask for the explicit right to do X in an agreement if there is zero possibility they will do X.
Why do these so-called "tech" companies need a separate document, a "privacy policy", to make users believe the company will not do X.
Why not include a prohibition against X in the terms of service.
To understand what the company does, read the terms of service. If the terms allow the company to do X, then assume the company does X. It is foolish to assume otherwise.
1. If there was a way to verify it, then the company would not need to make such assurances.
Google is terrible, but Google refused (they say so we don't know how true this is), to help the US government on spying, and Microsoft stepped in.
Bill Gates gets to sit with Xi Jinping and get called a "friend" and MS has exceptions in China unlike other companies.
Microsoft wants to buy ActivisionBlizzard, gets brought to the senate on the antitrust case, and the senators start barking against Sony instead, lol.
Microsoft is very literally all the US accusses Chinese companies of being for the Chinese government, now, so far it doesn't seem like MS has done that much with our data, but who knows what the future holds.
Isn't "we refused" what they would have said they did even if they haven't refused?
I can’t tell if this article is parody or not, and that’s concerning, haha
What? The chatbot is not your therapist. The chatbot is Microsoft. They are the same entity. Everything shared with the chatbot is shared with microsoft, just as every google search is shared with google.
Related legal determination re chatbots:
https://www.washingtonpost.com/travel/2024/02/18/air-canada-...
Perhaps some of the people who are commenting here don’t realize that there isn’t much of a way to opt out of these data “analysis” activities.
This has nothing to do with MS spying on user ai tool usage.
Admit it MS, you got hacked, service accounts and executive emails were leaked, along with who knows what else, and you "caught them" after damage was done.
I just don't see the harm. There are plenty of examples of privacy overreach on the internet, and Microsoft is guilty of some of them, but this particular thing seems fine to me.
And in this case, I think MS is doing the right thing! They shouldn't allow people to use these tools if they're going to use them to help them build malware. Certainly this sort of abuse-tracking should be narrow in scope, but it seems like that's exactly what it is, at least in this instance.
If you don't want a third-party reading your interactions with an LLM, build/train your own.
Eventually we may decide (legally, hopefully) that our interactions with AI tools should have strong privacy protections, but that has not happened yet, and companies are going to apply their usual privacy rules to these things: that is, not much privacy granted, at all.
I generally have a lot of respect for Schneier, but this sort of breathless, sensational "reporting" isn't doing privacy advocacy any favors.
Anything else is just stupid. Didn't we all learn on the playground that a secret isn't secret any more once you tell a friend?
Pretty much everything they do is either cloud connected inherently giving them visibility into what you do, or they feed constant telemetry back of what you do in one form or another in the os, office apps, whatever you happen to use they make, on whatever platform you use it on, including non-windows systems.
Really do you expect anything more of Microsoft at this point in history?
The OP seems to think that because the users region was recorded its spying on sessions. With a self righteous "I didn't read the TOS but This just confirms it".
Microsoft would obviously be recording region metrics, maybe even keyword/request trends. None of this information is identifiable and makes complete sense to be monitoring for improvement on a new product area.
A basic rule of thumb is to assume that virtually all commercial apps or digital tools (and many physical "smart" tools and products) pervasively, parasitically gather as much data about every aspect of your activity as they possibly can. This is the modern world's running business model: being constantly spied on by an ever growing range of objects and systems around you for possible monetization reasons at all times, even if you paid for them.
Governments make noises about opposing this but many of them tacitly love it because it spares them all the legal and logistical bullshit of justifying doing the same themselves.
Why bother after all? Now you can just buy or subpoena, or straight up illegally force (looking at you Snowden leaks) all of it from all the private entities that are given free rein to do what your "sadly archaic" laws still don't let you do.
Most absurd of all, that these same private entities make minimal effort at keeping their own monitoring a secret. The majority of their users literally don't give a damn and in fact freely hand over permission in a nicely, legally justified bundle. We essentially live in a Stasi, KGB or Gestapo functionary's dream of near panopticon.
Here's the full report for reference, but they don't answer how the data was collected or correlated to the specific "threat actors": https://www.microsoft.com/en-us/security/blog/2024/02/14/sta...
Does anyone else think this new APT naming scheme is terrible or is it just me?
All joking aside, and even as the article points out re: the T+Cs, is anybody surprised?
For example, they like to maintain that if you run an amazon competitor hosted on AWS, amazon insiders can't look at sales numbers in your hosted database. Or if you run a google competitor with e-mail hosted on gmail, that google insiders can't view your e-mails. And so on.
Personally I've always found these promises rather hard to believe, but a lot of people have a great deal of trust in them.
Using AWS when you're a competitor of Amazon seems like poor business decision making.
Or, better, who isn't a competitor and is also not at risk of becoming one?
It’s nearly impossible to install their products without opting into layers of spyware and adware cloaked in dark patterns and requiring extraordinary measures to back out of hidden spyware installs done without permission. Microsoft has always been a fairly malignant business, built in shenanigans of various sorts, and as the market perfects new shenanigan vectors they’re right there pushing the envelope.
Automatic sample submission
Send sample files to Microsoft to help protect you and others from potential threats.
We'll prompt you if the file we need is likely to contain personal information.
Even if you're not an adversarial government leaning on it as a hacking enhancement, it's only a matter of time till governments worldwide demand certain controversial conversational topics be reported to law enforcement. I suspect the Knight Paladins of Anthropic would be more than eager to further this greater cause for the Safety of mankind.
From the start, since all but a few services are using your data to improve their models there is already no notion of privacy. The ethical ones will eventually (if not already) report suspicious activity (as determined by law) to law enforcement and the less ethical ones will report all activity to advertising and insurance agencies.
Some already lobby to ban opensource AI because enhancing the learning rate and reducing the friction of gaining new information for humanity without controlling oversight will also enhance hackers or other bad actors ability to access sanctioned knowledge. They consider this a heresy and deem humanity at large incapable of responsibly handling such increases in cognitive ability. Only a few Adept can be trusted with administering AI. Truly, spying is among the more trivial concerns for the future of computing, given AI's compute heaviness and the amount of centralizing control it engenders by default.
This is exactly how the OpenAI blog post reads between the lines: “We work with the government to make the world safe. Or else.”
https://openai.com/blog/disrupting-malicious-uses-of-ai-by-s...
The actual “bad” tasks “bad guys” performed were no different that what one can accomplish with Google searches. But no one talks about stopping terrorists to use Google.
This seems to violate all kinds of boundaries between your personal and professional life.
Most people as they mature learn how important boundaries are to health and well-being, as a child or teen growing up, as a partner, as a parent, and with friends and community.
People don't need this mess too.
There is no clear way to opt out of this, to create and maintain a boundary and to say no.
https://simeononsecurity.com/github/optimizing-and-hardening...
There’s also a lot of extensive details of the dark patterns in the installers where no means yes with the privacy preserving option hidden behind rolled up options etc. Even then you need to do other patches to avoid telemetry and data collection. Even then you end up with ads surfaced in the start menu, etc.
Sounds like Microsoft sold the US citizens out to China and are now trying to play the victim card while adding Russia to the blame.
Say what you want about conspiracy theories but China and Bill Gates seem to be afraid of women to the point where they have chosen to try to eradicate them from society. Maybe the rumors are true about Bill Gates' attempt to make little girls infertil via vaccination.
Was the internet designed to be private?
But also, this is really the only way to actually improve the underlying tech. How do you know patterns of abuse? Normal use? Weird but non-abusive use? By collecting data on that usage and feeding it back into development.
You can collect that data internally or let users opt in; spying on users is absolutely not the only way to improve.
And no, it's really the only way to do it effectively. When you make it opt in or internal only, you get far too much selection bias to make the product good. That's just how it works with this tech right now.
And yet the poster finds Microsoft the bad guys in this story?
This is a story about our international enemies using advanced western tech to develop malware, with which to attack us.
It's right to be outraged, but not at Microsoft