CSAM Detection Technical Summary [pdf]

The final step of this is going to be detection of terrorist materials, then we will have privacy that is based on our jurisdictions definition of terrorism.

Your device provider(not necessarily Apple) will notify authorities when you score high enough on a threat model.

In just in a decade, for the US citizens that model could include belief in fair elections(depending on who wins, you can be a terrorist of the deep state or the far right), you can be a Chinese terrorist in the USA if the friction intensifies between countries, you can be financial terrorist if the relationship between Cryptocurrencies and the state sours.

You can be traitor terrorist of corporations, because they detected inappropriate use of corporate materials on your device.

You can be cultural terrorist in a culture obsessed with self preservation.

You can be infidel terrorist a muslim state with large enough market.

There is no need for hypothetical situations, you can be a terrorist or a freedom fighter today depending on your coordinates.

It is so easy to be a terrorist these days.

This is going to get very interesting when every authority starts demanding detection of their boogyman. Why bother with anti-espionage when you can tell apple to check who does have the pictures of your new shiny thing that was supposed to be a secret?

Apple seems to provide nice technological solution that looks promising to work fine as long as the authorities do good job in not becoming totalitarian crazy nutjobs because the technology can be used for any content, does not have a capability to ensure that it works only on child porn.

Apple is giving governments a content control tool that at this time is promised to work only for specific materials.

> Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices.

Is that a typo? What use is something that’s unreadable? I’m only half done, but it sounds like a bunch of hand wavy, AI, ML bullshit to me.

How do you match similar images without reducing them down to something that’s less than the original?

> Users can’t identify which images were flagged as CSAM by the system.

> If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated.

So the user can’t have information about anything, but they can appeal the decision if their account is locked? And locking the account might cause people to do all kinds of stupid things like resetting their device to get the account back or because they panicked. Then the only evidence is some opaque system that claims the user had illegal content. That’s scary.

Many years ago I knew a person that took a perfectly good PC to the garbage dump because of a scareware warning saying it was involved in illegal activity. Poor guy wasn’t very computer savvy. This will open up some excellent phishing opportunities because of the seriousness/consequences of the type of accusation and the legitimacy of everyone knowing it’s something that’s done now.

Edit: I finished reading it. If the photos are synced to iCloud I guess locking the account preserves the data. I still don’t like how cloak and dagger the whole thing is. No one’s allowed to know anything and all the average person is going to understand is that Apple’s system says this person is guilty, so they’re guilty. I think stuff like this would get a lot more scrutiny and less traction if it wasn’t “for the children.”

What if there are bugs? No one is going to be allowed to audit the system or see how it works because of the sensitive nature of the content. It’s pretty scary IMO.

> The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account.

How was that calculated? Prove it.

> The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair.

So is that the same kind of crappy AI that (incorrectly) detects suspicious activity on email accounts?

Many of these steps are spun as being for the users' privacy.

But they also prevent the operators of the system – Apple Inc, quasi-governmental non-profits (like NCMEC), & government law-enforcement agencies – from facing accountability for their choices. No one else can confirm what content they're scanning for, or what 'threshold' or other review applies, before serious enforcement action is taken against an account.

Also, Apple is quite vague about whether the claim "Apple manually reviews all reports" means Apple employees can view all triggering content, even if it was never uploaded to Apple servers. (How would that work?)

It would be trivial to extend this system to look for specific phrases or names in all local communication content.

How will Apple say 'no' when China & Russia want that capability?

In fact, might this initial domestic rollout simply be the 1st step of a crafty multi-year process to make that sort of eventual request seem ho-hum, "we're just extending American law-enforcement tech to follow local laws"?

It is unacceptable for this process to execute on my devices. I am very glad I own zero Apple products, and this has solidified my resolve to never change that.

I'll be watching for proliferation of this technique in other devices, and will promptly trash any that implement it. (I already avoid uploading unencrypted content to cloud services.)

In this imperfect world, privacy keeps innocent people free. Long live AES-256.

As a reminder for context, Apple already did do a server side check for iCloud photos [1]. I think that's missing from the debate, what's changing is moving the check to on device which many object to (and I and others have written enough about that).

To add to this though : I think they built an extremely complex - and hopefully perfectly robust - system to make the check on device. On top of the other issues, it's massively important said process is not exploitable.

The content of the NCMEC database is not public, one can argue for good reason, and if that effort to move on device lead to the database being leaked (and thus bad actors being able to "mark" content that they shouldn't share), that would be incredibly counter productive. Maybe only knowing that the database was updated is too much information to leak about this ?

I think that's an additional concern to the ones already expressed many times, and I sincerly wish Apple had kept the checks server side even more now, as I only see downside to not doing so.

[1] : https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...

The Big Brain Scheme Apple uses for their CSAM stuff. Summary for the lazy:

- this is going to be out in ios15

- Apple will build and push a database of hashes with every update

- these hashes will be used to verify against the pics before they are uploaded to icloud and generate a "voucher"

- once a certain threshold of vouchers is passed, Apple will be able to decrypt the pictures in questions (all this discussion seems to assume Apple does not have access to icloud pictures - so maybe they are preparing to rollout full encrypted backups???)

- the crypto stuff is interesting

- the scary part is that there is nothing preventing Apple to do the same thing with other thing on your phone (assuming there is a way to generate a good enough matching hash - I can see how this could be easily extended to basically anything that's stored on the phone)

This page [1] provides some more information and links to PDFs of several other related documents, including "Apple PSI System — Security Protocol and Analysis" [2] (which has lots more technical details) and some Technical Assesments.

[1]: https://www.apple.com/child-safety/

[2]: https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...

There's gonna be loads of fun picking this apart.

> Only another image that appears nearly identical can produce the same number; forexample, images that differ in size or transcoded quality will still have the same NeuralHash value

cf all the "calculated collisions" arguments already made

> Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the database of known CSAM hashes.

cf all the questions about that database of "official Badness"

> First, Apple receives the NeuralHashes corresponding to known CSAM from the above child-safety organizations.

sharing that authority out to who exactly i didnt catch the names in this document...

and also cf the already raised points about "won't $Nation want their own version of this with their own database of Bad".

Nations or religions, could Muslim countries keep their "no images of the Prophet" rules more strictly using these tools? would they not try?

On the neural hash algorithm, I presume it must be some sort of a standard algorithm since the hashes will come from third party organisations. Is it available as a library anywhere?

So are people really going to go to prison because of a flag that says they did something illegal? With no opportunity for discovery or to cross examine? No way to audit the software and the chain of events that led to the detection?

Just a question: if they turn these images into hashes, does that mean it only finds very specific images?

That is, if someone has some well-known image on their phone, then it can find it, but anything taken themselves is safe?

Doesn't that make this a bit pointless since I don't know anyone that saves images from the internet to their phone. That's weird, right?

For reasons unbeknownst to me at the time of writing (for it requires much more introspection), I think this would be OK if it were a service that ran only on icloud-stored photos— rather than apple devices across the board.

I’m not sure why this is the line crossed for me. Perhaps it is only because I willfully choose _not_ to store _any_ data on iCloud (or at the very least a minimum amount as possible). Even moreso, I’m a bit unsure of what that line even is.

Is it a software vs hardware thing? No, because one could argue the service is not Apple hardware-specific, but rather specific to the iOS software.

So perhaps the line crossed is this being an OS feature rather than a SAAS feature. It is unfortunate that the precedent has been set where I am OK with potentially privacy invasive features on SAAS products, but I suppose it is what it is.

From the document

>> The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. Descriptors are considered to be close to one another if the cosine of the angle between descriptors is close to 1.

Can someone explain what this means? Am I to understand that the algorithm plots image descriptors or hashes of them along a 2-dimensional circle, and somehow iterates until the way it places them on that circle results in similar images being closer to each other?

Seems if you don't use iCloud, you can opt out. There are alternative backup services available on iOS devices, like Backblaze. But does the OS let them properly integrate with the system?

This is a consequence of the composition of Apple's security engineering organization. If you look through the LinkedIn profiles of the people in the org that delivered this feature, an interesting pattern emerges: a significant number of them are ex-IC. Back in the golden age of early iPhone development, the org used to have almost full autonomy and was run by a bunch of Mac fanatics and UNIX nerds. Today the org is full of people who were spies in their previous job. It shows.

I was going to buy an M1 Macbook but I don't want it to execute any spyware processes. Is it likely that Macbooks will get this too?

A lot of that sounds like magic, most of us will take their word for it till it comes out and ppl like CitizenLabs takes a look

The bottom line here is Apple can look at your iCloud data whenever they want. They've written down a vague description of something that supposedly keeps them from doing it all the time, but they can unilaterally change the parameters whenever they choose to do so. They have thoroughly torpedoed their privacy story.

If you're an employee at Apple how do you even raise your voice of concern about this?

They're already calling people against this a screeching minority. Apple has created a toxic work environment and silences voices of concern by applying labels.

I am in favor of them checking messaging content for general nudity for minors but I seriously have my doubts about this CSAM system.

Your life will be destroyed by a false flag.