Microsoft's AI chatbot will 'recall' everything you do on its new PCs
theguardian.com
theguardian.com
I really worry that the financial incentives for this are all wrong. MSFT has a massive incentive to obstinately push AI interference into every element of their products. They must show traction to justify the continued exuberance around AI in the stock market. But a tool like this just has so much scope to make the user experience worse. I know what I want to do on my computer, do I need some AI clippy to ask if I need help.
Windows has been on a strong downward trajectory for a while now - and that's been a conscious decision. In the last week I can remember my computer - unprompted - telling me the pollen count, a football score, 10 teas to sip for international tea day. That the FTSE Auto Index is down 3.24% (just go that now). None of that information is useful to me, I feel a strong sense of hatred towards windows now and I will be dumping this computer as soon as possible.
But that's me, the user, I'm not who Satya Nadella cares about, Satya cares about saying "300% annual growth rate of AI usage" so that their numbers go up and to the right and I think it's going to drive me away from MSFT products completely.
“No” is not a vocabulary word in Microsoft anymore.
It's how I still don't have copilot on my Windows install despite the update bringing it to windows came to the US months ago. Or how certain employee tracking features in Azure/Office 365 are not available in EU.
I only continue to use Windows for work because it is mandated. All new work laptops come with at least one day of entertainment called "setting up". Where the myriad options are explored for the possibility of duplicity, breaking data collection and plain old contrarianism.
Not a long term user of those VMs though, so I have no clue how they behave over a period of months (etc).
E.g. I know I don't want to use Edge, yet it feels like one of those nightmares where you fight a thing that won't die. It keeps coming back and this is just one of those things.
Since I now played Fallout 4 without crashes on my Ubuntu system, I can't help but wonder if my future will be completely Windows-free at some point.
Except if it's a pattern that doesn't match what's perceived to be in the incredibly short term interests of Microsoft's shareholders.
At least you can hide those with a uBlock filter. You can't do shit if its baked into your operating system. (Aside from deleting the whole thing and installing an OS that respects your freedoms)
And a few months later, it’ll toggle to “yes” automatically, due to a “bug”.
And you can't even tell it "No", it's typically "Maybe later" or "Not now". When will "modern UX experts" understand that "No means No".
Honestly i don't think you can put blame on any single individual - including the CEO - and have things change, this isn't a single player (or even single corporation, considering Microsoft isn't the only one doing these shady things) game.
IMHO this is actually the biggest problem with modern application development, no accountability at all and everybody hiding behind somebody else instead of admitting failure, so all one can hope for is a poor compromise that's the average of many opinions (each single one of them even probably good, but not when mixed with random other opinions).
Funny enough, it's the same problem, everybody weaseling around a clear "No" ;)
Blaming a cog (for something the machine wants anyway) is tantamount to scapegoating. The machine will change out the part for one that more or less serves the same purpose.
Who do you think should be shamed for this? The developers who put it there? The UI experts who said where to put it? The UX experts that came up with this particular image as a way to maximize engagement? The product owners who told the team to add some extremely gross visual shock image to the app to boost engagement, and who approved that particular choice? The QA team that let it through? I'd say: yes, all of them.
You definitely can't put blame on a single individual here. There's lot of blame to be put on many individuals across the organizational structure. I picked an extreme hypothetical example to work like a contrast agent, to make it easier to see that there are parts of the org that are directly involved with some wrongdoing (vs. indirectly involved by e.g. setting incentives), and it's not absurd to want to put blame on all of them.
Regardless, though: at the end of the day blame must fall on an actual people, as it is people who make decisions and it is people who take actions and it is only people who can stop any of it. It is utterly ridiculous that even if we were to create a law that said it was illegal to create a dialog box that said "maybe later", somehow we would merely fine Microsoft and hope that disincentivizes the behavior while all the people who push on the law and continually test its boundaries in new products don't really have to worry about it themselves.
FWIW, I actually will say I am more than a bit sympathetic to the argument that we should just blame everyone vaguely involved to the point where if we meet someone who works for these companies we should look at them with some side eye... but, the reality is that some people at whatever company we are talking about (here: Microsoft) are obviously more to blame for whatever evil decision we are dealing with than others, and so your attempt to pitch a tent so big that anyone not just merely vaguely involved with a decision but somehow in the entire industry should somehow be deserving of equal shame--and so we should oh-so-conveniently just not blame anyone at all--feels like a disingenuous attempt at distraction.
What needs to be done is to address the incentives that cause the above.
The fact is, too many awful things occur because people are happy to hide behind companies and organisations and say they "had no choice". The idea of "just following orders", which has been a sad human tendency for millennia.
Yes, do call out bad practice when you see it. But please don't be hostile to (any/some of) the employees "who build these systems". They're doing their job/s. They're not responsible for the giant evil megacorp that employs them. They don't "empower" the corporation. It's obviously not that simple.
What is simple is that Microsoft doesn't treat Windows users like vermin because some random UX guy said to. It's the other way around. The UX guy got told what to do. The engineers got told what to do. They all got told what the software was supposed to do and they tried to make the software to do what it was supposed to do, because that's what their job is.
And for this one example, the imaginary UX guy actually made a more correct choice. The "maybe later" button is likely the best design given the constraints. The user isn't allowed to truly say "no, never", so presenting the negative option as "no" and then showing the dialog again a bit later would be a worse UX than the "maybe later" button which at least does what it says.
Speaking of giant evil megacorps, maybe direct your hostilities towards them instead. Your issue is with them, the company. It's their software/behaviour.
Of course they are! People aren't magically born hired to some giant company... they choose to--hell: they even fight to!--work there because they like making more money than they could at another job, in no small part as these jobs involve doing some evil things that extract more value from the world.
> They all got told what the software was supposed to do and they tried to make the software to do what it was supposed to do, because that's what their job is.
First off: who is telling them? Microsoft isn't some AI: it can't tell anyone what to do as it has no agency of its own... people had to make every single decision and take every single action that you are attributing to "Microsoft". They can't "all" be told what the software is supposed to do: someone--often someone low in the org chart, fascinatingly, but this sometimes could actually be someone high up--had to come up with the idea and do the telling.
Second: no... just because "that's your job" isn't an excuse to just do whatever your boss wants you to do, even when you disagree with it; that is a ridiculous level of blame shifting. If you don't want to be held responsible and you somehow can't quit then you should at least be helping in protest of whatever you are finding yourself doing.
> The user isn't allowed to truly say "no, never"...
I mean, why not? Well, because someone, somewhere--a person, with a name, whether they were high or low in the ranks--decided it was more profitable to take agency from users, and then a scant handful of other people were willing to go along with that decision in exchange for everyone being paid their share of the spoils.
(BTW: I would totally accept an argument that this button is a silly example to care deeply about. I'm not attached to the example myself: I was replying to someone else who cared about this and so am adopting this frame. There are much much worse decisions being made by Apple and Google these days.)
I'm not trying to make an argument about that feature. I think it's a good, overt example of Microsoft's attitude towards Windows users. There are many others like it and many are more and less offensive.
> I mean, why not?
You seem to have missed my point here (my bad). In the hypothetical scenario (the key points of which you are repeating) the imaginary UX team did their job well, by making the outcome of each choice more clear than they would/could have been and possibly even giving the user more than zero choice in the first place.
> Of course they are! [responsible for the company that employs them]
Of course they're not! It's obviously not that simple.
You seem to be making/repeating a "no one is holding a gun to their head" argument. There's not really any way to argue with that, because it's such an extreme point of view to hold. So I'll just say:
Yes, people make choices. I thought we all knew that. I also thought that we all knew that there are any number of complicating factors involved in any one of those choices. I think you should consider that not everyone has the power or opportunity to do whatever they want to at any point in time.
People should not do things they believe to be unethical. You do, as an employee, have the autonomy to say no to unethical tasks. You are therefore not blameless when you perform unethical tasks, because you chose not to say no.
I've quit jobs where I believed the project I was asked to work on were unethical, and that I had no other choice at the company. My first job out of university, I was asked to write code to cheat on an industry benchmark. I was 19 years old but I still somehow conjured up the courage to tell my manager that I would not feel comfortable working on that.
But you see senior engineers here with 20 years of experience just shrugging and saying "Well, boss told me to do it so I have no choice!"
The main point I was making is in the first paragraph. I was responding to what was essentially a call for random individuals to be identified and harassed because of their potential connection to some user-hostile software. I politely recommended/requested to not take such actions. I think if you have an opinion or concerns about a software/product/design/practice you should try to turn it into a meaningful contribution and express your thoughts in some way that you deem appropriate and helpful. If your preferred method of expressing your thoughts is shaming and harassing people, or other generally violent and harmful acts then you should keep your thoughts to yourself. The same applies to calling for others to act in such a manner.
> People should not do things they believe to be unethical. You do, as an employee, have the autonomy to say no to unethical tasks. You are therefore not blameless when you perform unethical tasks, because you chose not to say no.
You're not really disagreeing with anything I said (other than in intensity) but I'll respond to this directly.
People generally don't go out of their way to do things they believe to be wrong ("unethical"). People do tend to cross their own boundaries more easily when under pressure.
Different people have different levels of autonomy in life and at work. It's normal to find it difficult to say no to your manager.
Playing a part in some project does assign some responsibility to you. The smaller the part, the smaller the responsibility. Blaming an individual contributor for systemic problems with a project is unfair and a bad way to fix the problems.
Scale and context is important. Many/most things are issues of degrees. Applying your extremely rigid framework to a scenario with the scale of a single employee at Microsoft is bizarre/comedic.
That sounds an awful lot like "I was just following orders" :/
It has since become even worse. Forcing users onto cloud accounts, installing Zynga-tier crapware as part of routine updates and generally morphing into an advertising platform with a vestigial program loader attached.
The thing about the "maybe later" option is that it at least indicates what is going to happen when you click it. By the time that dialog was being designed, the 'aggressively recommend the new, worse Edge browser' "feature" was already set in stone.
I don't think it's believable that any number of these UX experts at Microsoft would have the power to change something as big as how the new, worse Edge browser is to be "recommended" to users.
I could believe an employee with a UX-related job title strongly recommended that all such choices are worded to explain what they do.
My SO after seeing me watching a video on this feature is now ready to switch her laptop and desktop over after checking that the two games she plays regularly are supported.
BUT. And that is a big "but": the first thing that greeted me on the joy of my new computer was a screen trying to gaslight me into creating a microsoft account. There was no visible option to just use a local account and I had to google a bit in order to invoke a terminal in order to get a local account.
I like windows for many things, including it's crazy good backward compatibility, especially when it comes to hardware. But shit like this makes it unreasonable to put a lot of trust into them for the long term.
And before we have to discuss the specifics here: this is just one example out of many where they don't treat users at eye-level, but like marks that you need to trick into liking you.
How about setting some positive incentives by showing me three really cool things that I couldn't do before if I use one of those microsoft accounts? Who knows, I might even make one.
But Microsoft seems to have forgotten how to use anything but bullying, monopolism and gaslighting to convince people to use their product.
Probably yes. :)
> E.g. I know I don't want to use Edge
They know too. The fact that they "ask"[0] you what you want (sometimes) and won't accept your answer is disrespectful, to say the least. They know what you want and they are constantly and actively working against you, to manipulate or con you into changing your answer. And that's when they don't just force something upon you outright.
Of course, they never intended to accept your choice. The software is acting on behalf of the vendor, not the user. Other software that does this is often called malware.
[0] Calling what Microsoft does "asking" is being extremely generous...
But sure, assuming minimal Wine wonkiness (nor pray not, some interference on WINE deveoplment by MSFT), I'm sure many gamers can transition to Linux and keep their library intact.
Game support via Proton has been relatively good though.
Well LLMs have already stolen everything you ever wrote or posted online.
We told y'all to advocate for a very strict regulation of these things.
But no.
Instead: It's amazing. It's world changing.
Yes. It's going to change the world. For much worse. If you thought Citizens United damaged democracy just wait. Elections will be won by those with the most compute aka the most money everywhere.
People will die of this bullshit. I can't foretell how but they will. I mean, I haven't foreseen AI generated mushroom books but I am absolute certain this technology is deadly. Just wait.
Everything AI generated should come with an unskippable, extremely obvious huge warning of being AI generated and free of facts. If the AI companies won't add these then we should force them by regulation. But no. Hype instead.
I kind of expected that someone will steal something I write or post online, so this doesn’t particularly bother me.
It's a truism the Internet doesn't forget.
But copyright is copyright and copying some work just because it's published on the Internet is illegal, it always was and still is.
> But no.
Simply legislation doesn't work that fast. Building up consensus about what needs legislation and how it needs legislation takes time.
> Instead: It's amazing. It's world changing.
People don't all think with a single mind. The people who find it amazing are not necessarily the same who complains about it.
My two cents: no, the refusing option will be deep in the UI and surely you will need to use a combination of CTRL, ALTs, etc to make it visible before you check it out.
Many times it is more difficult to (securely, optimally, etc) configure a Microsoft Windows desktop than a Microsoft Server with Active Directory, and MS Exchange.
You might be surprised. Satya reads Hacker News and there is an internal Microsoft initiative to promote Microsoft tech here[0]:
> In fact this morning I was reading a news article in Hacker News, which is a community where we have been working hard to make sure that Azure is growing in popularity and I was pleasantly surprised to see that we have made a lot of progress. In some sense that at least basically said that we’re neck to neck with Amazon when it comes to even elite developers as represented in that community.
https://www.microsoft.com/en-us/investor/events/fy-2019/earn...
Data mining the users of the most popular desktop operating system is a too huge profit opportunity to miss for ethical reasons.
There are not many companies which I’ve grown to immensely dislike over the years like Microsoft. I started my career as a .NET developer and huge fan of their products, buying into all their crap from all the way from Silverlight to Windows Phone, WPF and even owned a Surface Pro 3 tablet/laptop but today there is really rarely a company that makes me reck as much as Microsoft with their product announcements. My opinion of this company couldn’t be any lower and I hugely judge any developer working in Windows. I know I shouldn’t but I can’t help myself that unless they are a game developer there is objectively speaking absolutely no reason for a developer to work on a Surface device or Windows.
The Candy crush and other crappy apps/games in combination to being a full fledged spyware OS with adware on top is already enough to make me run away screaming.
Android and Chrome OS aren't really a replacement for desktop OSes, and Google is even worse in spying users anyway.
So uuh... where are you shipping software then?
Now imagine it happening for literally every interaction. Opened a context menu? Oh give me a second to check the LLM server for useful actions I can offer you. Want to close a window? One sec just wanna analyze the content first to see if you reealy want to close it.
I am so happy that most gaming is now possible on Linux and I dont have to use Windows at all.
We'll probably never get such an emphatic apology, instead:
- there will be no response (same way mirroring a Disney movie in the Vision Pro is just a dark rectangle)
- it will tell us to just see with the administrator (the closest we have to "go fuck yourself", as we'll be the administrator)
- it will throw us a link to a help page with platitudes about how we're mostly free to do whatever we want, and ask us if the page was helpful to solve our problem.
- we'll be asked to repeat the request indefinitely with no response on the result.
In practice, some White Hat will find a way to disable this (yes, even if it's running locally).
1) You like all of this "AI" "stuff" and you use the newest version of the Windows Operating System.
2) You just want an Operating System that works so you continue to use Windows 10.
3) You use Windows within a corporate environment where the Security and IT teams neuter it with Domain settings to disable most of these "AI" "features".
Until Oct 14, 2025: https://learn.microsoft.com/en-us/lifecycle/products/windows...
If you felt so inclined you can also pay Microsoft for support after that date.
Or Microsoft might also extend support based on public demand, given that Windows 10 still beats Windows 11 in install base.
There was a time when this was true of Windows 7 and we all know how that went down.
Though I have serious doubts that an ordinary version of Windows will ever get released. I mean one without bundled OneDrive or "AI" services and without a dumbed down desktop experience.
Don't give anyone any ideas.
…what could possibly go wrong
Cloud 2 = Cloud 1, but delegate more compute/logic to edge/local computers to reduce cost. [1]
Cloud 3 = Cloud 2, but edge/local computer has hardware specifically made to do the compute, store intermediate data, and send the results to someone else who might give you something in return for free, and possibly make you pay for it later.
Marketing can call it Cloud <3 (heart). Some might say </3 (broken heart). [2]
> [~10 years ago] Smartphones began incorporating AI accelerators starting with the Qualcomm Snapdragon 820 in 2015. [3]
> An AI accelerator, deep learning processor, or neural processing unit (NPU) is a class of specialized hardware accelerator or computer system designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and machine vision. [3]
[1] https://en.wikipedia.org/wiki/Volunteer_computing
Now please log in with your Microsoft account.
Your account is suspended until you provide proof of purchase or pay full retail price for 90 cans, plus 200% surcharge.
I predict a near future where AI agents will do the same.
Europe starting to look very prescient here.
BTW: if you uncover a willful violation of these laws at your company (preferably with the knowledge of a director), you can whistleblow to the SEC (and the relevant EU authorities at the same time), and the SEC will provide you strong whistleblower protections and hopefully you can walk away with millions of dollars in whistleblower awards.
To my knowledge the EU doesn't provide the same kind of juicy whistleblower incentives, but willfully violating such a strong EU law would also trigger SEC enforcement, and they're the ones who will pay big bucks.
To my knowledge the largest SEC whistleblower award was $279 million.
I encourage you to reach out to a lawyer, e g the non profit Whistleblower Aid.
Source: [1] https://www.zuckermanlaw.com/sp_faq/sec-enforcement-actions-...
If the US pass a GDPR-like legislation, try to get monetary damages and make it a business for lawyers, because in Europe the money goes into the public coffers.
Discussion at the time (featuring very similar privacy concerns): https://news.ycombinator.com/item?id=33421751
Whose new PCs?
How they probably intended it: Microsoft-supplied PCs (as the vendor of the operating system and/or hardware)
The snarky second meaning: Microsoft literally owns the PCs (a PC running Windows/Copilot is Microsoft's property)
My preferred interpretation: The new PCs belong to the AI chatbot. The fact that the chatbot has access to everything you do on its new PC is unsurprising.
Definitely screenshots will be seen by team who will label the data for further LLMs training/fine-tuning.
The above is only half joking as someone was literally convicted for reposting a meme on Twitter in the past few months.
I need to start asking if I will need to use a windows computer for work on interviews and end it there if the answer is positive.
Where do they store all this data on my local PC? Some news items mention screenshots every couple seconds, for example.
Will you be forced to pay for extra storage for a feature you don't want or need?
Will Microsoft send me an extra SSD on their dime on request? :)
We focused on LLMs that can reason while still remembering key facts exactly.
But I like that the concept is catching on in the AI community. Go Satya.
LLMs are powerful, but unlike humans, their weights are fixed during inference and cannot be used for working memory.
Computer memories are abundant and perfect. It's not unreasonable to store your entire life in all modalities in a personal computer of the near future.
AIs with that ability should exceed human performance on knowledge tasks.
It can NOT reason and exact facts are simply not a property of LLMs.
https://www.inc.com/kit-eaton/why-a-former-white-house-cyber...
Miss a proper office and games tho
Yeah right!
I've already swapped to Fedora, a few more other projects and I'll start on the kids and wife's machines.
The only windows devices left will be corporate devices.
The details so far suggest Recall will run locally, offline.
The demo vid showed that you can select apps not be indexed in Recall… so, not “everything”
From the article: > Microsoft promises to protect users’ privacy by giving them the option to filter out what they don’t want tracked.
This line here suggests that you can opt out of specific tracking, but says __nothing__ about where the model will be stored, trained, etc.
https://www.microsoft.com/en-us/event
“We leverage the power of the NPU to ensure content stays on your device, which makes the experience fast and maintains your privacy”
Now whether you believe that or not is another matter. But that is the claim at least
What happens in 10 years, now, I make no claim.
https://arstechnica.com/gadgets/2024/05/microsofts-new-recal...