Apple to scan iPhones for child sex abuse images | Hacker News Reader

Apple to scan iPhones for child sex abuse images

bbc.co.uk

112 pointsmonkeydust89 comments

Earlier discussions:

Apple's plan to “think different” about encryption opens a backdoor to your life – https://news.ycombinator.com/item?id=28079171 – Aug 2021 (748 comments)

Expanded Protections for Children - https://news.ycombinator.com/item?id=28078115 - Aug 2021 (353 comments)

Apple plans to scan US iPhones for child abuse imagery - https://news.ycombinator.com/item?id=28075021 - Aug 2021 (368 comments)

Apple enabling client-side CSAM scanning on iPhone tomorrow - https://news.ycombinator.com/item?id=28068741 - Aug 2021 (716 comments)

A better title would be: Apple to scan iPhones for banned images.

SwiftOnSecurity:

> Just to state: Apple's scanning does not detect photos of child abuse. It detects a list of known banned images added to a database, which are initially child abuse imagery found circulating elsewhere. What images are added over time is arbitrary. It doesn't know what a child is.

(from https://twitter.com/SwiftOnSecurity/status/14233832560037478...)

Nice. Well, Apple had me consider a switch from Pixel phones when they prioritized privacy over Facebook. Since I don't want anyone scanning and deeming my pictures "acceptable" I just stick with Google. And they were that close to become an acceptable big tech co regarding user privacy...

> Since I don't want anyone scanning and deeming my pictures "acceptable" I just stick with Google.

Don't Google already do this for images you store in their cloud?

   yes.

[0] https://www.theverge.com/2014/8/5/5970141/how-google-scans-y...

[1] https://torrentfreak.com/google-drive-uses-hash-matching-det...

Sure, but so does icloud and onedrive. And while i don't like that either, at least one can opt out of it. The only way to realistically opt out of this new system is to not use an iOS device. And i'm just wondering how long until they want to scan your mac as well.

> The only way to realistically opt out of this new system is to not use an iOS device.

You can just not share the photos to iCloud, the exact same way you'd opt out of it happening on iCloud, no?

I suppose you are right about that actually. Makes me wonder why they chose to implement scanning of files destined for icloud (locally) when it's already a feature of icloud?

Scratch that. I think i get it now. They now have that capability BUILT IN locally, which might mean it can possibly be used via remote command on-demand (on for example non icloud pictures). Which is kind of scary.

That's why I hardly have any pictures there. Except stuff I share spontaneously. The vast majority is on local drives. I guess I'll have to do that in general going forward then...

EDIT: Just checked, I don't backup pictures from my phone in the cloud. At least that's what my Google camera app insists on.

So if those things are equal you’re going to switch to an iPhone then?

One of them is scanning on device. The other isn't.

> One of them is scanning on device [...]

                                 [...] when the photo is about to be uploaded to the cloud.  If you don't use iCloud Photos, it won't be scanning your photos.

> about to be uploaded to the cloud

it's scanning ON your device. This means Apple now has the capability to scan and find files on your device. Which means they have opened the pandoras box of over-reaching government requests.

Same here. Despite the atrocious UI of iOS, I've been slowly and reluctantly considering iPhone for my next phone because of the good privacy news. This news kills any consideration of an iPhone for me. I'd rather go back to a dumb phone than let a company scan my phone at will.

I'm currently considering a Pixel phone with CalyxOS. It might be the good in-between choice.

They also use on-device machine learning to detect pictures (and videos?) containing any kind of 'undesirable' content.

"Messages uses on-device machine learning to analyze image attachments and determine if a photo is sexually explicit."

Furthermore: "This program is ambitious, and protecting children is an important responsibility. These efforts will evolve and expand over time."

Per: https://www.apple.com/child-safety/

The road to hell is paved with the best of intentions.

How likely are perceptual hashes to give a false positive? If I take a picture of a tree, how likely is it that a few pixels are going to line up just right in a hashing algorithm and say it might be child porn? How likely is it that law enforcement is going to understand the limitations of this technology? How likely is it that the judicial system will understand the limitations of this technology before issuing a search warrant?

I can see law enforcement showing up at my door one day demanding to have a look around, and I would have no idea why they’re there, but they’ll want to look through all my personal belongings.

Worse yet, I might come home from work one day, see my windows broken, see my place has been ransacked and my computers are missing. I would call the police to report a burglary only to hear that I’m under investigation and they need me to give them the key to decrypt my hard drives.

I feel like I need to do some risk analysis on my digital life. I need my phone for 2FA apps for investment accounts. If I use a dumb phone, I’m stuck with SMS, which is easy to hack. But if we use iPhones, there is a small but costly risk of a completely erroneous investigation. How do you balance these without knowing the probability and impact of the later?

EDIT: OK I didn't read the article, that's my fault and thanks for the clarification. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud. --------

At least it's a lot more possible to mis classify a photo than a sms message. Plus as I mentioned in another post, what if you have a young kid and you happen to shoot a photo when he or she is not fully dressed?

Why would that photo match a list of hashes of known CSAM though?

This technology at Apple is very much about matching hashes.

Thanks, original post edited.

> If I take a picture of a tree, how likely is it that a few pixels are going to line up just right in a hashing algorithm and say it might be child porn?

p = 1.0.

(This is a sports in the machine learning community: https://openai.com/blog/adversarial-example-research/ .)

It is more likely someone malicious will send you a picture of a tree that has been doctored to match the hash of a known offending image.

Of course you will be off the hook after a long embarrassing investigation.

and after your children have been taken away preemptively by CPS possibly for months and given to a foster home that may actually be abusive.

This pushed me over the edge to get the last remaining thing I use my iPhone for working on my Pinephone: maps. Marble (from KDE) is really good and even has offline routing support. I think I'll buy a GPS unit just in case (I was planning on getting one for a boat, unfortunately I haven't seen any that do both sea charts and turn by turn but I'm all ears if people know of any.) My iPhone is formatted with an empty battery and in a drawer now. I worry for other people though, the privacy implications of this are insane. This is a huge misstep even for Apple.

I’ve got the feeling that they’re already doing this on their servers for photos that are uploaded to iCloud (which I believe is pretty standard for services like this). By moving this scan to the devices themselves they can offer end to end encrypted storage without losing the ability to scan for child sex abuse images. Hence, I wouldn’t be surprised if they launch end to end encrypted iCloud storage within the next year.

Yes they already seem to scan iCloud:

https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...

But I seriously doubt they will make iCloud end-to-end encrypted. If they wanted to do that they would have done so already. And why not announce it at the same time, which might give them an overall PR win? But we shall see I guess!

Moreover, if they make it truly end-to-end encrypted, they wouldn't even need to scan on the server side (since they'd only be storing random bytes from their perspective).

That's pretty much the theory of a lot of people on this. On-device CSAM scanning as a trade-off for making iCloud backups, photos etc. e2e encrypted.

I’m a nudist. I’m wondering whether we’ll be a persecuted minority (as opposed to merely derided comic relief).

Judging by how Apple has handled the App Store you'll certainly be reported.

What about people with young children? I know my parents have a good number of 35mm shots of me in the bath with my brother and sister when we were babies.

> What about people with young children?

It's not matching an AI's interpretation of "is this a naked child?" though - it's specifically only matching perceptual hashes against a known database of CSAM percpetual hashes provided by NCMEC which will then be manually reviewed if there's more than a threshold of matches.

So we should assume any and all of our photos are probably going to be looked at by some random people?

What's stopping said people from leaking photos?

> So we should assume any and all of our photos are probably going to be looked at by some random people?

No? Not unless they're matching the perceptual hashes of the CSAM provided by NCMEC and in that case, sorry, my sympathy does not extend that far.

If anything, this makes it harder for random Apple people to look at your iCloud photos if moving the matching step down to the phone means iCloud photos are going to be end-to-end encrypted, right? (Which is not a given, of course, just a theory that some people have about why it's moving down to the phone level.)

> Not unless they're matching the perceptual hashes of the CSAM provided by NCMEC and in that case, sorry, my sympathy does not extend that far.

You trust automated systems more than I do.

If we don't own the phone, we have no basis to insist on a search warrant before a search. So moves to make it obvious that Apple really owns the phone and we're just pushing buttons on it, undermines any efforts to establish legal precedents around searches and seizures.

>If we don't own the phone, we have no basis to insist on a search warrant before a search.

Thats not how it works. Your house is still protected from search and seizure even if you rent it.

EDIT: OK I didn't read the article, that's my fault. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud. ----------

I want to add that this may be a serious issue for people with young kids. Imagine you shoot a video/photo of your kid with little dressed on. Now what happens? Are we supposed to fully dress kids because we have no idea what Apple will do after the scan?

This.

Not this. Read the damn article, people. It clearly says it's matching pre-determined hashes, nothing to do with AI. You're making this worse than it has to be.

OK I didn't read the article, that's my fault and thanks for the clarification. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud.

It checks images against an on-device database of known CSAM, your photos should be fine.

OK. Another argument: what if a virus download the pic for you?

At that point hope you have enough money to hire a well known forensic analyst to prove you were infected with a virus. Back in college I used to repair pcs and there was a virus a few people got that turned their machines into torrent nodes for vast hentai collection... Now if that had been child related and I had to report them I always wondered what were odds prosecutor would choose to charge them or actually take time to see if it was fully automated or manual and they were viewing the content as it went through.

FWIW Microsoft already does this for photos you store on OneDrive: https://www.theregister.com/2014/08/07/ms_tip_off_child_abus...

That's great but irrelevant. This discussion is about scanning them locally on your device.

Indeed. Thanks for pointing this out. Scanning on device is a different story.

Well, it's semi-relevant, isn't it? Because it's only scanning photos that are being uploaded to iCloud Photos - they've just moved the scanning portion down one level.

If your photos never go to iCloud, they won't be scanned.

So what's the point of doing this client side then? It seems like they're deliberating paving the way for doing this on all client side material.

> So what's the point of doing this client side then?

Not a clue. Maybe they're going to make iCloud Photos fully E2E and this is the only way they can keep the CSAM scanning alive? Or maybe they just want to avoid it arriving on their servers at all (but I don't think anything has said matching CSAM photos will not be uploaded?) Or they just want to save some CPU on the iCloud side by punting that down to the phones themselves?

(Or, yeah, this is just the precursor to doing more client-side scanning and the slippery slope cassandras will be proven correct.)

Powerful corporations aligning interests with law enforcement by assuming certain LE roles seems a recipe for corporate dystopian oligarchy.

This opens, for Apple, a catalog of answers for "How can we make a buck," and none of the answers will be friendly to privacy or freedom.

So wait, I have pictures of my baby son’s poop to send the pediatrician — which show what will eventually be his naughty bits, as he becomes sexual (in due time, years from now)

I actually don’t have many nudeish pics of him because it’s very cold right now, but we take dozens and dozens of daily pictures so — but in the summer… when maybe he’s going to the kiddie pool grandpa has…

I guess it's just a disguised protest against the government by Apple, that they have to give up their stance on privacy and make the world know.

Yes, children is so precious and helpless. I think in order to protect them you all have to give up your liberty, so to protect the future of human race.

Time to use a dumb phone. I realized recently that I can uninstall most apps on my modern phone and still live without any issue. That's the privilege of immigrating to a foreign country as friends and family now don't expect you to reply as soon as possible.

I wonder if this is secretly brilliant stunt.

Blast on every channel message that your product will actively look for children porn store on it. Stir great debate.

And then quietly drop the proposed feature.

Your product will be viewed as unsafe to use by the pedophiles maybe even whole apple ecosystem will.

Let me get this straight- the iPhone will now scan your iCloud account and compute hashes for images. These hashes are then compared against a known database of child abuse images. When a certain threshold of "hits" is reached then steps are taken to notify authorities?

There seems to be little chance of false positives here. I keep seeing references to machine learning algos and that's not what they are doing at all. I do still lament losing yet more control though. A device that I buy should not be working for somebody else, for better or worse.

CSAM is not new and is already in widespread use. The difference here just seems to be that your device itself will sell you out.

Edit: Follow up question- They run this on images before they are uploaded to iCloud. Why has Apple chosen to do this on-device rather than just run it in iCloud? They say for privacy reasons:

'Apple’s method of detecting known CSAM is designed with user privacy in mind.'

Isn't this very very silly? You just need to change one pixel and the hash will be different.

No, perceptual hashes don't work that way. It's a comparison of similarity to a target, not just making an MD5 hash of an image.

Here's Apple's documentation on it : https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

Oh nice, didn't know about this.

Very helpful. Thanks! Will give this a read.

Thats not how it works. The fingerprints/hashes do not need be exactly the same. Just close enough. The fingerprint/hash is designed to not change much if the pictures are similar.

Simplified: If you draw a circle and i draw a circle, their fingerprints will be closer then if I would have drawn a rectangle.

Based on an article I found from 2015 on MD5, it takes about $0.65 in compute and 10-12 hours to make small mods to an image to achieve an arbitrary hash. I dont know this hash used here, but would it be reasonable to create a 'make small changes to this image until it has this hash value'?

If so, there will be false positives but they will be intentional. Black hat can send a collection of 100 modified but innocent meme images with nefarious hashes.

Does it matter that they are only doing this for images uploaded to iCloud?

So would this make a good vector for black mailing? Also I guess many parents might have a lot of pictures of their small children playing around naked.. at least where I live.

Looks like that concludes my otherwise very nice time in the Apple ecosystem. Luckily Linux is coming along well both on phones and the new Macs.

what stops apple from checking if you have any content they don't like of like non-pc memes and such and flag you in some databse for futher monitoring/enquiry by pc police?

that is the big concern that in future you share pepe frog meme or in say russia a pride flag and it gets marked as terrorist image and they report you to _____ or block you from posting on ______

Slippery slope...

Stuff like this is triggering my whataboutism alarm as not a day goes by without the self proclaimed leaders of the "free world" attacking certain governments on nothing but suspicion and "intelligence".

Before 2013, the more cynical ones among us were simply branded conspiracy nuts and crackpots.

If anyone believed Apple's lies about privacy, that's on them. The solution is to buy more smart stuff from American companies. They are the only ones that can be trusted

It does not seem to be scanning phones, but stuff uploaded to iCloud, which is completely different. The article gives the idea that your device will be scanned.

>"Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes," Apple said.

It is scanning phones, but only files that are about to, but haven't yet, be uploaded to iCloud. That scan is happening on the phone itself.

That actually make it more okay with me. Apple can't have child pornography on their servers, that would be illegal. However, the fact that they are doing the scanning one the device could indicate that they don't have the ability to do the scans in iCloud. Presumably they can't read even read the images once stored in iCloud, so they have to do it on the device.

I don't know if that's the reason, but seems like a reasonable guess.

Apple actually isn't legally liable for what users upload until it's reported to them. And they are capable of doing the scanning server-side, since iCloud doesn't use end-to-end encryption.

> since iCloud doesn't use end-to-end encryption

Interesting. They say they do, seemingly for many things, though not all[1]. Do you have more info?

[1] https://support.apple.com/en-us/HT202303

You are right that some specific features on iCloud do have end-to-end encryption (only those listed under "End-to-end encrypted data" on this page).

But the majority of users' sensitive data is not included in that set of features. For example the Photos (what's being affected here), Drive, and Backup features don't use it. Note that any encryption keys backed up using iCloud Backup are therefore effectively not end-to-end protected either.

Somewhat misleadingly, this page indicates those features use encryption both "in transit" and "at rest", but Apple controls the encryption keys in those cases, so they are actually not end-to-end encrypted.

Here is a report indicating that scanning is already happening on the server side: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...

Here is a report about how the FBI specifically pressured them against adding end-to-end encryption to iCloud backups: https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...

>>Before an image is stored onto iCloud Photos, the technology will search for matches of already known CSAM. Apple said that if a match is found a human reviewer will then assess and report the user to law enforcement.

>>Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes

Exactly, only "private" cloud data will be scanned instead, which is industry-standard practice for any self-respecting cloud provider anyway. It's a wonder how Apple wasn't doing it already.

In any case, this will be automated, rather than some poor Tier-1 pouring over iCloud Photos.

So only the guilty (and the false positives) would worry.

This is false, the scanning occurs on the phone. Plus, as has already been discussed at length, the NCMEC database is loaded with false positives.

The "nothing to hide" argument tends to fall apart when the database being used against you is full of legal imagery (which often isn't borderline or pornographic at all -- some of the flagged images literally don't show people).

Slippery slope to non-CSAM material? That ship has sailed already. The databases are a mess. From day 1, it detects non-CSAM.

And whatever would make us believe that as users we should expect a difference between a company's servers and the same company's devices?

Would the device belong to us, or be covered by some misguided sense of ownership and privacy?

The garden was already walled, and the name on the gate wasn't the user's to begin with.

Now, where's my rusty yet trusty Nokia? Oh, wait, it doesn't have VoLTE...

> So only the guilty (and the false positives) would worry.

If you truly want to "protect the children" you should have no issue for the police to visit and inspect your, and all of your neighbors houses. Every few days. Unannounced, of course. And if you were to resist, you MUST be a pedophile who is actively abusing children in their basement.

You're not guilty, are you?

My point exactly. What is privacy in the face of CSAM, after all.

/s (if not already clear enough)

Innocent until proven guilty implies no false positives. What happens if I get arrested because of a false positive? What happens to my life because there will always be that doubt from everyone?

My social life is crippled for the rest of my life because of a false positive. Which can happen to anyone. Which means everyone should worry.

I am not sure I fully understand the concern people are raising.

Everyone already doing this, Google Photos/Drive, Dropbox, Youtube, etc. The only difference is Apple just started doing it?

Edit: people are downvoting for asking a question...

It's done locally now. Hardly the same thing. And sounds extremely illegal to me

Apple are proposing to do it on your phone. There is no possible way they are not going to do it to everything on your phone.

That's a big policy change, since they're circumventing their own E2EE using their super-admin powers to do it.

The barrier when uploading to unencrypted services is that you are at least aware you're granting access. This is invading the phone to do it anyway - that's quite different.

But more importantly: when flagged this is going to send the results to human reviewers. Which has not been elaborated on: there's no way human reviewers can screen things without being sent a copy of the image in question. Which means, a neural network system is going to randomly send your personal images to human reviewers - and it will, largely, be false positives.

Note that Apple are not discussing anything about the reliability of this system: have they run it against a sample set of normal images? How many did it flag? Because, if they give a number, it'll be pretty easy for people to realize that some % multiplied by the number of phones and average number of images per phone, and I'm willing to bet what you get is: the system will false positive at least (and probably more then 1) personal photo per iPhone user, and send it to human reviewers.

And that might get people's attention: certainly more then just, unfortunately, us techies.

EDIT:

And let's talk about these human reviewers: these are not random citizens seeing something and wondering if they should be concerned. The context these people are going to be given is possible child abuse image. This is not a neutral review process - at all.

The concern is I bought an iPhone based on the promise of privacy of my device. Now there is a monitoring tool on my phone looking at photos for child porn. I get a false positive and some rando from Apple is now looking at my personal photos. Why?

Normalisation of this invasive behaviour is not okay. Apple specifically argued they are not Google and do not invade your privacy.

Is it relevant that other companies were doing this already? No. Apple wasn't doing this, so people invested in using Apple as a digital platform for their digital lives instead of the companies you mentioned, suddenly we are stabbed in the back by them.

You make it seem like it is weird to care about sudden local government corruption just because there are other countries that already were corrupted.

The other consideration here is that Apple demands total control of your device with the promise of security and privacy in return. If they don't follow through on that then why should I surrender so many of my fundamental rights as a user to them?

Reply on news.ycombinator.com