Apple's bright idea for CSAM scanning could start persecution on a global basis

Microsoft Zune. The Super League. 47 Ronin. Wonder Woman 1984. Google+. Big companies make big mistakes. No matter how many focus groups you collect, no matter how many phone surveys you do, things can go wrong.

Why is that so difficult to imagine? Apple's security model has always been "just trust us and don't question it", questioning their own practices themselves just has never been done.

That's already what's happening with any other Apple software, their own services are off limit of their model, explicitly excluded & explicitly trusted. This opinion is also reflected in their security threat document they published, they never talk about themselves being in the list of potential threats.

That's just in the continuity on how they usually work.

Edit: It's also notable that news of this was leaked before Apple was able to officially announce anything. This means that Apple's marketing department was not able to control the tone/narrative as well as they normally do. The first thing many people heard was "Apple will be scanning your phone" without any of the nuances that came later.

Then in 2020, there was the EARN IT Act proposal, which nearly passed and would have required scanning for CSAM on pretty much every online platform that wanted Section 230 immunity.

Apple puts two and two together, realizes Congress is concerned about CSAM's spread and isn't interested in changing, and still wants E2EE on iCloud. OK, put the scanning client-side, then the way is paved for E2EE iCloud because the FBI's biggest argument against E2EE is neutralized, and so is Congress' argument for EARN IT (which would basically have banned E2EE).

Did somebody raise the concern of push-back? I'm sure. But the moral questions around CSAM are something that was settled long ago internally. When I was at Twitter fighting abuse, the CSAM stuff was a separate group. My boss called them The Department of Mysteries because we almost never saw them or spoke to them. It was led by a serious person, an ex-FBI agent or something like that. They did what they did and we were all ok with it and grateful for it, because that shit is horrific and we didn't want it on our platform and we didn't want to have to deal with it ourselves.

My cousin was a PO for sex offenders, and one of our regular discussion topics at family reunions was how sex offenders were way more technologically savvy than a state parole department. How they really needed more help in making sure offenders weren't reoffending while on parole, while also not forcing them to just not use computers and phones altogether. If even I've heard this, I'm sure that Apple execs have heard it from law enforcement a zillion times.

It's also clear Apple put a lot of thought into addressing the privacy concerns for this. Technologically, it's sophisticated, impressive.

So I can easily believe the people at Apple said, "Sure, there are reasonable concerns, but we think we have addressed them." And that they're surprised by the level of sustained pushback.

One of those two statements I disagree with.

Apple exiting the privacy industry would look like this to me: "we've decided from now on that, like almost all other cloud providers, we'll give ourselves access to your stuff for (ahem) legitimate purposes".

Not like this: "we'll implement a neuralhash on the client device rather than on the servers, and then do a cryptographic private-set-intersection protocol with them on the server".

That's a lot of cost and effort to prevent themselves, as a company, misusing the CSAM detector for other purposes. If there are government agencies involved, Apple is also making sure that they can't just use this as a backdoor to get access to everyone's files, it's as if the government said "we need to prevent child abuse, give us a backdoor" and Apple went "ok we'll give you a small backdoor that's ok at detecting abuse images and nothing more" - if the government was expecting to use the backdoor for more than this, they'll be disappointed.

(I'm pretty sure they have other backdoors already, by the way. My guess would be a zero-day on the baseband processor firmware.)

I'm not saying I agree or disagree with Apple's latest move, but "exit the privacy industry" feels a bit a strong statement to me. You have less privacy than you did three weeks ago, and an option on even less privacy in the future (but then again Apple could just change the T&C), but you're still better off than with competitors that offer similar functionality.

I think that a number of folks at high levels in the SV executive suite have accepted a manufactured consensus along with their peers in Washington, and that consensus is something like: "people don't care about the privacy of what's on their device and they'll put up with anything to stop CSAM, even if it means scanning personal backups and local files (as opposed to shared files.)"

This seems like a reasonable thing to believe, since server-side scanning of (mostly shared) files has been going on for years and nobody has pushed back very hard on it. But what I think the consensus missed is that the reason for this lack-of-pushback is that nobody in the wider world had really been asked to weigh in on it before. It was something that a few elite tech busybodies were aware of, and most people accepted the idea that providers needed to check out photos that lived (unencrypted) on their servers. Apple accepted this logic and extended it unthinkingly beyond shared photos to unshared private photo libraries on the user's personal device (even if they are staged for backup as part of the iCloud Photos synchronization service, which is just a policy choice.) This was a second mistake because it assumed that because users mostly ignored the scanning of shared server-hosted files, they had somehow given consent to having their private files searched on their device. I don't think they had.

Overall, this announcement is the first time anyone has attempted to have an actual public debate to see how real users feel about this kind of surveillance, particularly automated surveillance of private photos (and an automated system with potential flaws.) Apple's mistake here was to assume that their user base had already given consent -- when they'd just never been asked. It's a very human mistake to make, frankly. The question is whether Apple will listen to their users or if they'll double down and push this through against their users' pushback. I can forgive Apple for misunderstanding their users once, but continuing down this path will be a lot harder to understand.

ETA: To illustrate how much more pervasive Apple's surveillance is than the standard (ignoring the PSI protocols), consider this quote from an EU Parliament briefing: "Others, such as Dropbox, Google and Microsoft perform scans for illegal images, but 'only when someone shares them, not when they are uploaded'." (I can only trust that this is factually true.) In this sense, Apple's move to scan all photos in your library is a significant functional escalation.) https://www.europarl.europa.eu/RegData/etudes/BRIE/2020/6593...

Especially “all content must be reviewed for child porn before publishing, or in real time if streaming”.

It now seems not at all implausible that Apple’s half-baked attempt to scan everything for child porn is due to this too.

They have serious problems admitting they did something stupid historically. Butterfly keyboards, reliability issues, you’re holding it wrong etc.

After having read a lot of internal Apple emails between executives[0], I find it extremely easy to imagine that they were completely dumbfounded that the rest of the world did not see things the same way they did.

[0] https://twitter.com/TechEmails

They already scan and classify ALL your photos locally, not just the ones destined for iCloud, that's a widely touted feature.

Now, you might say that China would force Apple to scan for things anyway - and they might. But at the same time, China could have ordered this at any moment in the last decade. It's not like the Chinese government wouldn't force this tool to be made anyway.

https://money.cnn.com/2018/01/10/technology/apple-china-iclo...

Not really. Virus scanners don't snitch on users to governments.

Companies try to do stupid things and end up with egg on their face all the time.

They'll backtrack from this and people will forget it ever happened in a few months.

The second market is bigger and less discerning. Its a purely economic decision: come and bring your kids to our nice safe child-proof walled garden.

There is also the parallel requirement in places like China and Russia to police what people have on their phones. This move clearly differentiates Apple from other Western tech companies and maybe protects them from bans in big and fast growing markets.

Remember those NSO iOS security revelations from a few weeks back? It might be quite possible that the government(s, specifically the US) are handling some information over in exchange for this. It would make sense for Apple to reach out to the government and get information about some of the vulnerabilities in exchange for brownie points, because this CSAM system is totally controllable by Apple while unknown security issues aren't.

Now whether Apple designed this only with CSAM and actually ignored/forgot its possible issues.. that's another question.

> why they enables CSAM in the first place?

Apple's privacy measures, such as not scanning your Cloud photos, is what helps enable CSAM.

Sexual abusers very often take photos and often upload these to their communities, and Apple has given them a secure device with which to do that. This is becoming increasingly widespread — the number of reported CSAM material grew by more than 50% last year, to nearly 70 million images and videos [1].

Due to Apple's privacy, the numbers look like this: Facebook reported over 50 million combined images and videos, Google reported 3.5 million, Dropbox, Microsoft, Snap, and Twitter over 100,000 images and videos. Apple reported only 3,000 photos in the same period, and no videos. [1]

Sexual abuse is experienced at some point in childhood by around 1 in 9 girls, and 1 in 53 boys. 93% of perpetrators are known to the victim. In 2016, CPS substantiated or found strong evidence to indicate sexual abuse of 57,329 children [2]. That's CPS, meaning in the US alone.

> They must have known the consequences, PR and otherwise.

The PR consequences of that have gone largely unnoticed, interestingly, and it's more so their attempts to curb it that are getting flak from the media.

I suspect this is because we can't create statistics on what we can't detect. There can be no "Apple is enabling, and allowing to continue, the sexual abuse of 50,000 children a year."

[1] https://www.nytimes.com/2020/02/07/us/online-child-sexual-ab... [2] https://www.rainn.org/statistics/children-and-teens

Then in 2020, there was the EARN IT Act proposal, which nearly passed and would have required scanning for CSAM on pretty much every online platform that wanted Section 230 immunity.

Apple puts two and two together, realizes Congress is concerned about CSAM's spread and isn't interested in changing, and still wants E2EE on iCloud. OK, put the scanning client-side, then the way is paved for E2EE iCloud because the FBI's biggest argument against E2EE is neutralized, and so is Congress' argument for EARN IT (which would basically have banned E2EE).

Furthermore, "right to closed borders" ? You previously invoked "free movement" yet there is also a right to closed borders? It appears you've just cloaked the same tired red team talking points in the language of freedom. Please, as a libertarian, stop trying to use freedom to justify what is a highly authoritarian movement. You're doing freedom no favors.

Mixing together a bunch of weak arguments does not create a strong argument.