This simple site is a far better demo and explanation of the extreme danger of Apple's proposal than any of the long articles written about it.
Thank you for caring enough to put this together and publish it.
Thank you for caring enough to put this together and publish it.
We should cool down a little and wait for the real system, before expecting anything from them. Let’s see then how bad it is, now we are just victims of speculation.
However, this is something which have been planned for years. Bad PR won't undo this, and stepping back is not happening overnight.
There is so much misinformation in this subject, because it is really hard to see the bigger picture. We are in this situation, because of the leaked information. Was it intentional for political reasons, who knows.
Apple is definitely going to announce some E2EE features very soon, such as backups in the next event. It is very hard to invent better system, than they have released now, to enable E2EE and apply CSAM scanning.
When you step into the Apple system (which is quite closed, and nobody really can make extensive review for it), you give all your trust for Apple. Whether this scanning happens on your phone, or in the cloud, it does not really matter, as long as the end result is the same (same files are scanned).
How about the abuse of the system? Prior on-device scanning, government could ask Apple to add feature for telling, if users have Winnie the Pooh images on their phone. (Could you spy for us?) After adding the on-device scanning, now they ask Apple to add this hash dataset to find Winnie the Pooh images. (Could you spy for us?) The question is still the same. Does some technical change affect for the managers and the chair?
How about forging CSAM images to make FBI to show up in the neighbors door? Not going to happen easily. The only problem here is to brute force their human review process. In that case, it is further double checked by NCMEC. But still, at first we need some hash which exist in their database, and secondly that image should be uploaded to iCloud. And not only once, 30 times with different hash.
That's what Apple wants you to think, and they're very successful at marketing, but it's not really true. The jailbreak scene is not what it once was, but there are innards to Apple devices and services. Apple engineers put their pants on one leg at a time, the same as everyone else.
[1] https://www.theguardian.com/technology/2019/jul/26/apple-con...
It seems to be getting a lot of attention because it's an easy target, but it doesn't seem to be a very meaningful one given the system they say they use.
It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)
Apple isn't exactly known for software transparency or allowing software that monitors iOS internals.
Many reasons this is bad.
Obviously the "Slippery slope" argument around it scanning more in future and being abused by the state.
There is the possibility that its exploited/bug that sends data when it shouldn't or falsely reports you (not a collision). Eg. a bug that reports all photos as bad instead of listening to results of scan.
Obviously the possibility of collision sucks. But thats somewhat constant across implementation options.
It wastes your CPU resources and network resources to do something that only benefits apple.
It's a bad precedent because other companies will likely do this now, repeated the above arguments with increasingly less trustworthy companies.
It seems that this system might be extremly resistant for collision to prevent them ending up to human review.
https://news.ycombinator.com/item?id=28305946
> It wastes your CPU resources and network resources to do something that only benefits apple
Okay, how practical is this problem? iCloud already scans your files to know whether they are need to be synced. And this scan applies lesser times, only once before upload per file.
Encryption does not mess with the filesize, so we are talking about few bytes of extra metadata per image here. Regular image with raw format is tens of megabytes.
> Encryption does not mess with the filesize
No but giant bloom filters take storage.
Building these types of systems is antithetical to a free society.
Can you predict the US' political climate in twenty years' time? No? Then don't build this.
And how do you propose that all of these evil regimes are going to get their images into the NCMEC database? The hash DB will only include photos that are in NCMEC and a second countries’ CSAM database.
And it will be trivial to verify that the hash DB is consistent across different countries.
All they have to do is say: in order to sell iPhones in this country, you must do what we want. Apple will capitulate.
I am russia.
I take actual child porn from my vast kompromat databases (perhaps sent to me helpfully by facebook), or have my agents make some more themselves. They have many talents.
I use adversarial modification to make the child porn match many images of gay pornography popular with the peoples of my country. I add these modified images to my child porn to my databases and also ship it off to the relevant agencies in other countries. It's obviously child porn, so of course they add it.
Apple staff forwards me matches, the images look pornographic and they hit the database. Failure to report child porn you discovered is a felony, so they will error towards reporting.
If for some reason apple doesn't forward on enough of the matches, I hack their severs, kidnap their staff, or simply order them to provide the data they are already collecting (on penalty of not being able to sell in russia anymore). I can continue to use the pretext of searching for child porn to do all this with a smile.
I think this is all obvious enough, and I'm sure the people who work in this business are smarter than we are, have capabilities we can't imagine, and can come up with even better attacks.
[0] It's federally illegal for anyone to have CP on their system, or to view it.
[1] So the workaround is displaying a distorted low-res version instead https://www.apple.com/child-safety/pdf/Security_Threat_Model...
[2] You can replace CP with anything your oppressive regime is trying to snuff out.
This is not correct. Having a few will put those photos through a second, blind hashing algorithm. It wouldn't be flagged for human review unless the images in question passed the second stage and, even then, you wouldn't have any of the information for those hashes to match.
Having a few will have the flagged photos sent to their servers, now liable to be obtained by external forces. Then processed by a second visual hashing algorithm, liable to the same type of collision attacks, using obfuscation as its promise of security.
> Once Apple’s iCloud Photos servers decrypt a set of positive match vouchers for an account that exceeded the match threshold, the visual derivatives of the positively matching images are referred for review by Apple. First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.
So to recap, this second blind hash you're talking about is run _after_ the server has already received and decrypted the visual derivatives (what else would it be hashing?).
Perceptual hashing is highly susceptible to collisions. I'm not sure what you're talking about with "hiding a different thumbnail through resizing". All you need is nude-looking colors in your photo to let the derivative look like credible CSAM. No resizing tricks necessary.
The downside it that this makes the human review process tricky. Because there is, understandably, no actual CSAM image to compare the visual derivative with. Images that look similar enough (contain nudity or a child in a swimsuit) may get past the human review process.
Having a image sent to LEO is effectively being accused of a crime.
What prevents a judge from approving a search warrant on the basis of an image that matched a hash? The answer is, of course, “nothing”. What guarantee do we have that these matches aren’t used as a signal to some other automated extra-judicial search? Again, none.
None of this is fantasy. Take a look at the no-fly list and and no-knock warrants and ask yourself if you really want an automated system making accusations to LEO.
e: clarified that images are sent to LEO, not just hashes.
"Those targets will not meet themselves"
"What if that person has CSAM but tried to obfuscate it. Would you forgive yourself if you let it go and it turned out you were wrong?"
There's a fair point in there about public defenders not meeting that bar, at least in terms of the quality of their representation given the amount of work they have, though.
Lawyers aren’t always doing their maximum. It doesn’t matter that they get paid, you also have to be among their top important customers, just like aircon repairmen.
Also, the picture is not CP but by that time, the phone is already entirely searched.
Law enforcement has no incentive to exonerate you, but they have tremendous incentive not to waste everyone’s time, including their own, on fake cases with little chance of conviction.
Sadly, in the US prosecutors regularly mislead the court because our adversarial system of law is seen by many to obligate them to zealously prosecute their case up to the boundary of the law. (That the prosecutor should conceal evidence, mislead, exaggerate, right up to the point where any further and their actions would be an unambiguous crime)
As a result, there have been child porn prosecutions over totally legal material rescued only by the actress in question showing up in the court. ( https://www.crimeandfederalism.com/page/61/ )
At the end of the day, Apple's decision to reprogram their customer's private property to snoop on their users private files and report on them creates a risk for those users. The risk may arguably be small, at least today. But it is a risk none the less.
We don't need trillion dollar corporations snooping on users phones like masked vigilante crime fighters. Batman is fiction, and in even the Dark Knight when he spied on everyone's phones it was clearly portrayed as an erosion of his moral character out of desperation.
People are routinely arrested over the results of field drug tests despite their high sensitivity and high probability of false positives.
I dunno man, I’m not to worried about this chain of events.
When photos are uploaded to places like Google images they are scanned for this material.
Apple is currently only processing images that are being uploaded to iCloud. So, these are photos people intended to upload to them. This means it's not unwanted upload of photos to Apple servers. That is happening anyway.
Note, I think scanning should happen on Apple servers and not the phones. This is an invasion onto the devices in my book.
I wonder if this is a way to scale the system and reduce their servers and power use.