Windows AI feature that screenshots everything labeled a security 'disaster' | Hacker News Reader

Windows AI feature that screenshots everything labeled a security 'disaster'

theverge.com

73 pointsalwillis42 comments

I am once again amazed at the inability for anyone within Microsoft to prevent or halt the development of things that nobody wants, needs, or could reasonably expect to be a good idea.

Actually this is something everyone will want. Not by Microsoft, not built into Windows, but this is something that has to come in some form.

I don't want it.

I'm increasingly convinced that at this point it is, for these companies, mostly about putting out the word "AI" in press releases a lot until such time as the markets lose interest; any actual utility is completely beside the point.

> things that nobody wants, needs, or could reasonably expect to be a good idea.

Maybe for you. Sincerely, the NSA. /s

Sounds like great tech for forensics, surveillance, and legal discovery.

And employers.

I don't think employers are going to be that interested in this. It's not like there's a shortage of tools to monitor employees usage of company owned devices.

Rather this seems like a giant security hole, I'm almost certain that my company would permanently disable anything like this on our computers and I'm sure we're not the only ones.

Its going to be great at one thing: employers will think they can use it to execute more control over micromanaging employees. But will end up getting hacked and releasing all the corporate emails exposing all the illegal shit by management

I suppose it depends what side of discovery they're on.

Oh, ha, if a company is under some kinds of investigation, could they be instructed not to disable this record-every-little-thing feature on any company Windows computer?

Finally, it's the year of Linux on the desktop! :)

Interesting. Could be used to create fake alibi too - i.e. I will use innocent browsing of social networks logged by this feature as an evidence that it was not me who got access into server which should not be accessed in that specific time.

The thing is: Even if one can disable this "feature"... Who guarantees that Microsoft will not enable it again by some update or suddenly introduce a remote version which is enabled by default?

Microsoft can not be trusted anymore because regulators failed in the past to properly punish companies for dirty behavior.

You mean a feature which records all user activity into a global, well-structured, explicitly searchable structure might be a ripe target?

Obviously unforeseeable and not at all a feature implemented rashly in a clever but fragile manner.

PSA for how to disable Co-Pilot (and Recall) on Windows.

https://www.reddit.com/r/Windows11/comments/1bh7n4d/how_do_i...

What’s the point really? Microsoft has a habit of re-enabling all telemetry with Windows update which is really annoying to block or disable. They will do the same with Recall and just say it’s a bug that turned it back on.

I haven't used Windows in ages but there used to be popular tools that were used even by not-so-very nerdy, more average folks that would search your PC, indicate all the problems they found with wasted disk space and security, and offer fixes.

It still goes without saying that without a resounding "I understand the risks and still want to use this inherently insecure feature" things like these should not be enabled by the OS manufacturer / vendor.

[dupe]

Actual article: https://doublepulsar.com/recall-stealing-everything-youve-ev...

More discussion: https://news.ycombinator.com/item?id=40540703

Apple is working on something like this too I bet but with Siri + OpenAI

Wasn’t there an open source software (called “Rem”) that someone posted about here not that long ago? It was basically the same thing as this Windows feature but it was for Mac.

It's not a good idea, but I don't see how security is a big part of the problem. In particular crying foul about the data being stored in the clear seems silly. How would encrypting it help, when the key has to be right there anyway? Is it that much worse than all the other sensitive data also sitting in the clear in, you know, regular files?

What's the plan, encrypt the data a public key where the private key is locked most of the time? Require a passphrase any time you want to read or process it? Does anyone think that UX will fly? Maybe there's some TPM magic that makes it work without also making it trivial to steal for an attacker who already has arbitrary code execution, but no one is citing that so far. I don't see the better design that everyone seems to think is obviously there. (Besides the obvious Just Don't, which is fair as far as it goes but has nothing to do with crypto)

Just for example: It could be stored encrypted with the user's credentials. So when you want to browse "Recall" it might pop-up requesting a fingerprint, password, token etc.

Then it decrypts, presents to the user, and closes the handles when you're done. In this way, the "key" isn't present -- the user has it!

That's quite a different security situation to it just sitting around in plaintext on disk. Certainly there's a number of Windows and MacOS features sitting behind authentication already, from a UX perspective.

I don't think users would go for that UX, so if you're committed to building this feature, I don't think you can go that route without basically accepting that no one will use it. I suspect MS wants this to be a lot more casual than the other features behind authentication now. (Like I told akkartik, I'm open to the argument that there's no way to implement it securely and with an acceptably easy UX, but I don't see people actually making that argument.) Maybe they'll prove me wrong by doing something like that, though.

The way I read these arguments is: the lack of security adds to the case against this product. The reviewers aren't saying there's a self-evident solution, just that there are self-evident problems. How did this get through review? So it's not that there are two separate arguments, but that the security issues strengthen the main argument that this is a product that shouldn't exist.

That's a very charitable reading. In my reading, the dismissive tone of the "security" arguments implies there's something obvious that MS should have done better specifically security-wise. If there's no such better implementation, then security has nothing to do with the argument, and they should stop talking about it. If they wanted to make a more nuanced argument about how there's no way to make a secure implementation that provides the smooth UX people will actually use, I would respect that a bit more, but I'm not seeing either that particular claim or the evidence to back it up in the mainstream commentary.

I don't know what else you've read, but in the context of OP in particular it's not clear to me what seems dismissive about the "security" arguments.

It's stronger in the initial coverage, I believe by the guy who discovered the story and the immediate commentary on Mastodon. The Verge is a more professional outfit so they're more subtle about it, but it's still there in phrases like "That could make it trivial for an attacker" (emphasis mine) and "There are clearly some obvious holes". Tone is hard to prove, so I'm probably not going to try any harder than that.

I can totally believe random microblogging takes are stupid, but that doesn't seem worth arguing about. Microblogging is just capturing initial reactions.

> "That could make it trivial for an attacker" (emphasis mine)

It _is_ trivial, right? The attack _is_ trivial. It's the solution that isn't. OP isn't claiming that. Seems reasonable to me.

I went back and reread https://doublepulsar.com/recall-stealing-everything-youve-ev... which is the first link in OP. Again, it doesn't have much to say about encryption at all except to point out that hey, the protections Microsoft is claiming are underwhelming. Without those protections the cost/benefit balance is way out of whack.

While I'm not a fan of the almost forced OOBE setup, I really don't understand the complaining regarding security. It's again another case of if you can run code on my computer you already own it. As Raymond Chen put: "you’re already on the other side of the airtight hatchway. And you’re bragging that you can do something annoying like a denial service, apparently unaware that being on the other side of the airtight hatchway gives you the ability to do far more interesting (and threatening) things".

This kind of "vulnerability" has been a constant topic of his blog in the past:

Reading about the details of this, it really looks like a feeble attempt to jump on the AI bandwagon. Such a shoddy design could easily be implemented on top of any linux desktop distro, and much better. Also isn't it amazing that despite being fully in control of the OS, the code of this recall function will burn GPU cycles doing OCR on images of the screen?

looks like a coop project, or a friday afternoon tech demo. hard to believe its on by default

I like the idea of Recall. But its a Microsoft with 'lets hide/remove these settings from user' bullshit, so...

> the feature is enabled by default when you set up a new Copilot Plus PC, and there is no option to disable it during the setup process

It's coming to every new Windows 11 powered PC in the future so this will be harder to ignore.

Could this be the best contender to the stupidest idea on the last 10 years by a big tech corporation?

it's like a keystroke logger on steroids

Microsoft payed a shitload of money to tech oligarchs for "AI" and is now desperately looking for a use case. And this is the best they could come up with. The future sucks. We gave it up to some scammy tech bros and the tech illiterate management caste.

Dystopian hubris

Screen shot that password creation screen

I’m not at all a “year of the Linux desktop guy”… at all… but seriously, I’m so very glad I dropped Microsoft last year. Even the idea here was a bridge too far.

This feels like some crazy joke taken from a fictional alternate universe. Or that I am dreaming. Slapped myself and checked - nope this is true reality.

I love the feature, it finally convinced me to give Lunix another try!

Recall doesn’t perform content moderation, so it won’t hide information like passwords or financial account numbers in its screenshots [which it then stores in an unencrypted SQLite DB]

I predict this thing will be big, like Scripting Host and Outlook were in their day /s

Windows 10 is a disaster, this is a catastrophe!

Reply on news.ycombinator.com