Except that every time they get a positive, the device will upload the image to Apple, where at some point an actual human will have to check it.
Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This is just a slight twist on the common prank of making people Google terms related to CSAM.
Not only will that cause Apple to violate the privacy of innocent people who were just saving cat pictures from Facebook, it will also require a lot more manpower than Apple is probably anticipating to keep up with the amount of false positives that will come in. And that isn't even considering unintentional false positives.
I don't think Apple is planning to start sending out "oopsie we violated your privacy and looked at your memes/family photos" mails to affected people. But it might come to that too once this planned practice is tested against current or future European privacy laws (and such a notification would be the bare minimum in my opinion).
That won't go over well once the first people share such an experience.
1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers.
2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review.
3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced.
4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.
Apple claims without evidence that there is a second secret hash which is run serverside. Perhaps it's just Microsoft PhotoDNA (which isn't secret anymore), perhaps it's just another copy of neuralhash (which might share the same collisions)-- it isn't currently known to the public.
It is, however, known to anyone inside the project at apple and anyone involved in the abuse image databases at a minimum.
I'm of the opinion that any security properties of this supposed second hash should be disregarded in public analysis of the system-- for all we know it's just a "return true". Other people make a reasonable argument that this additional function makes some of the attacks people describe harder. In practice it probably does, but security analysis should concern itself with the worst cases admitted by the known system.
This isn't a reasonable stance, it's just a tactic of poisoning the well such that you can never lose the argument.
I refrained from outright accusing you of bad faith when you claimed that the example image was obviously manipulated, though I confess that was my instant impression. I'm sad that you haven't extended me the same courtesy.
It's considered a bad practice and insecure for systems to keep a cryptographic _algorithm_ secret. In a cryptosystem all that should be secret is the key itself which is small and uniformly randomly generated and by construction cannot contain any weaknesses that could be uncovered by inspection (as it's just uniformly selected out of the space of possible keys).
This doesn't at all apply to a perceptual hashing system: the system will certainly contain weaknesses which could be found by chance or by limited black box interrogation, but it could only be cheaply reviewed for flaws with direct access to it.
The fact that secure cryptosystems by construction protect the keys from leakage through usage is part of the reason that it's okay to stuff all the security dependency into the key.
If you used HMAC with an unknown key and allowed me oracle access to see if inputs I sent in matched, I couldn't learn anything about your hmac key (unless the underlying hash was broken). But if we didn't know the weights of neuralhash and I got match oracle access to it and access to a few matching images (maybe 1 per bit of hash), I could likely the entire set of weights just through interrogating an oracle that gives a pass fail. (Also the papers that successfully design adversarial inputs for unknown networks, simply by designing adversarial inputs for a set of entirely different networks)
Moreover, this second perceptual hashing system requires that Apple share the system with third parties-- they must share it with e.g. NCMEC and the agencies of other governments because it is a felony for apple to knowingly handle the unlawful images themselves so they cannot produce the database themselves, it has to be produced by NCMEC that has the special statutory carveout. By contrast, in a standard public key encryption or authentication scheme your private key is shared with no one ever. That makes it much easier to keep secure (though keys often do get leaked and stolen).
Finally, if I steal your pgp key or whatever, it effects a single use or a single user. Most cryptosystems use as many keys as reasonably possible so the loss of one has the limited impact-- and they can be replaced with another uniformly generated random key if there is a concern for a key's security. But there is only one second image hash, they likely cost millions to develop, so the leakage of it would be much more damaging to the system compared to leaking a key.
I'm perfectly willing say that we have a reasonable disagreement on value of the 'secret' additional hashing scheme. The role of obscurity in security systems has been debated for decades in forums far beyond here-- and we're not likely to add more to it in this discussion.
But please don't accuse me of poisoning the well because we have opposing perspectives on this broad subject. I think I've adequately argued why your comparison point isn't a great comparison. We might not agree but we should agree that neither of us is taking a disingenuous position.
Security by obscurity in 2021?
The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.
That's good, because they weren't arguing that NeuralHash is infallible, but that its fallibility does not lead to the consequences outlined by the GP.
> The fact is that a 'first line of defense' that is trivial to fool is useless.
Useless to you, but not to apple. It reduces the set of pictures to process from "everything" to "those identified by the algorithm as designed, plus artificial false positives". It's impossible for the second set to be larger than the first, so it's by definition not useless.
This is why it's strange how people are focusing so hard on the weaknesses of NeuralHash when what we should care about is how well the system as a whole works.
I presume Apple didn't break the law and morals to construct the hashes.
I am being facetious, I know, but my point is that humans' morals and ethics are incredibly widely diverse ― especially if we're talking about the "effective" instead of "proclaimed" ethics (i.e., the values people actually follow instead of values they believe/claim to follow).
This might actually be brilliant and we might end up thanking them for it. It might pretty much destroy this filter.
> This is just a slight twist on the common prank of making people Google terms related to CSAM.
And here is the problem: there is one failure mode of this and that is if a bunch of ordinary people get jailed for innocent memes before the public gets whiff of whats happening.
So if they do they should probably be targeted as a waterhole "attack" at Apple execs and pro censorship politicians.
Apple employees reading this: think very closely before downloading wallpapers the next few years.
That's a very optimistic view.
In reality, they need to automate something while they never get any feedback whatsoever if their efforts are successful, let alone worthwhile.
Crooks that attack Paypal at least know when they succeed - they see the money. In this case, not only they won't see the money, they won't know if what they're doing is "overwhelming" the system or not.
Complaints about the proposal seem to overly focus on the reliability of singular aspects of the automated detection system when this has already been planned for - additionally those comments seem entirely unaware of the threshold technology in play.
Intentional false positives require knowledge of a hash. And even that isn't enough to get a human to scan your image. You would need to acquire a library of A1 CSAM images/hashes. You would need to generate images which match both NeuralHash and the unspecified secondary hash algorithm which runs on Apple's cloud servers. And then you'd need to repeat this trick a sufficient number of times to trigger human review, at which point a few people (probably some minions in a developing country) get to look at the images for 1 second before pressing the "not CSAM" button and having the images sent to a technical team in Cupertino for analysis.
Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery.
Intentional false positives are irrelevant because there are, depending how you count them, between two and six other layers which would need to be thwarted or exploited in order for intentional false positives in NeuralHash to result in any adverse outcome.
So if the purpose of the system is not to catch persons dealing with CP, what is the purpose? Breaking user's privacy?
We have a history of evil intentions masked by fighting against CP, drugs, money laundry and crime in general.
I see no reason to think that general intelligence, technical awareness, utility maximization, impulse control, or risk-reward calculation wouldn’t have similar distributions such that plenty of people would end up in a set prone to share risky material on such channels.
One possibility is that these databases are stuff full of material which isn't actually CSAM. There is no accountability for the content of the databases.
Considering the infinitesimal number of prosecutions compared to the claimed number of hits, we should hope that this explanation is actually the truth...
A system that detects Covid-19 in infected people and doesn't detect it in healthy people, where the errors of both types (false positives and false negatives) are irrelevant, is not only relevant, it's a very well designed system.
So, you're accepting that Apple's approach is useless for anyone who's "determined" but you still seem to be arguing there's some merit in this complex "architecture" of surveillance.
So the idea is that mass surveillance is great even if it's effect is scaring criminals while it actually only catches people at random?
We've arrived at the opposite of the common reprise of law. No longer "It is Better that Ten Guilty Persons Go Free Than that One Innocent Person be Convicted" but "It is better that Ten Innocent Persons Be Dragooned Than that One Guilty Person Go Free"
It's a tough line for them to walk.
Personally I don't see any substantial moral difference to what Apple was planning versus what Google and Facebook already do, aside for the somewhat icky aspect that some initial CPU cycles are occurring on hardware I own.
I'm equally concerned that a purported need to scan for CSAM could be used by legislators as a way to convince the population that the Government needs more encryption backdoors. Personally I'd rather have my photos inconsequentially hashed than have encryption legislatively broken.
Actually, you've made quite a few claims, including some that even ineffective surveillance is good 'cause it makes criminals worried. But anyway...
that I understand why Apple doesn't want CSAM on their servers, while also reserving the ability to implement end to end encryption on photo galleries in future.
Given you've admitted Apple's filtering is ineffective, you have admitted that criminals could put CSAM on Apple servers.
But weird thing is "end-to-end encryption" is entirely useless if everyone is being surveilled before the start and after the end. Sure, Apple wants to be buzzword compliant with privacy and entirely surveil it's users in reality. I suppose I too can see how that would be appealing.
The phrase "all I'm saying is" is an idiom, not a literal assertion.
> Given you've admitted Apple's filtering is ineffective, you have admitted that criminals could put CSAM on Apple servers.
Yes, I did. And I responded to that already. I'm not interested in engaging with "if it's not perfect then it's useless" type of arguments.
Having illegal activity on a major cloud service makes it harder for those services to offer robust encryption without law enforcement pressuring legislators to mandate backdoors.
But in the US it isn't just a backdoor, it's an indirect violation of the user's 4th amendment rights. If the government ordered apple to perform this scanning, the scanning would be unlawful and the result would be inadmissible in court. It is only lawful because Apple performs it voluntarily (or at least that is the pretext maintained in court-- repeatedly Apple staff have maintained to me that the government is requiring them to do this). The government should not get the ability to execute unlawful searches simply because it's able to use soft power to coerce user trusted third parties to perform the searching on their behalf.
The code/network detecting this will be run on the client. So yes, a 'hash' or an equivalent will be known.
You would have to hope that the secondary system is robust enough to not fall victim to the same manipulated image. And it really is on Apple to first prove it isn't.
In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash.
Edit: I am almost entirely wrong, see the comment reply from krisoft. Apologies!
No. You are wrong. They do hash each image and they do upload the hash derived safety voucher always. Here is the Apple technical paper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
There is a fairly complicated cryptographic algorithm which, according to apple, lets them open the safety voucher only if there is a match with the database.
The user devices only have a cryptographicaly blinded version of the hash database which they can’t turn back to real hashes without apple’s private key. This means that the plain neuralhashes are not available to the end users.
They use cryptography to conceal the hash database from the users to guard it against accountability.
Because of this, if they stuff the database with political or ethnically targeted images no one will be able to tell, you just have to trust that they haven't done that.
Knowledge of a matching hash isn't a big barrier for another reason: Matching images are widely available. NCMEC reports that they had over 20 million matching images reported to them last year. It's a very reasonable assumption that some 4channer will helpfully troll the internet for likely illegal images and put up a large database of likely matches.
False positives can also go the other way: Modify illegal images to match existing lawful images which are already in circulation. One could likely take legal hardcore pornography, crop it to make it impossible to identify the ages of the talent, add captions to falsely claim that they are under-age, tweak it to match targeted images, and submit it to the databases.
The encrypted list is stored on the devices for security and performance reasons: otherwise it would be too easy to ship people customized lists with arbitrary images. You can't just rapidly revise the list without undermining the privacy protection here.
Also the space of modifications is gigantic, for any given image there should be millions of differently hash variations which can be reached without major visual damage. (at least for neuralhash the figure is billions)... so you can't just stick all of them in the database.
You could stick widely circulating variations in the database, however... sure.
Are you saying the hash DB can't be updated or that already uploaded photos won't be compared to hash updates?
It can be updated-- at some cost. But if it is very frequently updated then the protection against receiving a targeted list is lost.
To state it another way: Users ask "Why won't the government order apple just gives me a list of the hash of every political image on the internet in order to expose my files?" and the answer is that the database is part of the OS, common to everyone, updated infrequently, so it can't be used in targeted attacks. That protection is lost if the database is frequently updated.
It's also true that already uploaded photos won't be rescanned, but that wasn't the point I was trying to make.
My comment was pointing out that the well known fact that the neuralhash is highly vulnerable to preimage attacks (intentional false positives) also means that is is highly vulnerable to intentional false negatives. Changing the 96-bit hash to a specific value while minimizing the visible image change is a LOT harder than just changing any single bit, but the harder task has been amply demonstrated.
Neuralhash is one component, yes, but it is a limiting component. If a slightly tweaked image has a different NeuralHash it will never be detected by their scheme.
The App Store is presumably full of easy to use apps which can store photo galleries in a separately encrypted vault. This is where people will go after being shooed away from iCloud, not image manglers.
Cops plant illegal drugs on suspects all the time. Why not this?
BANG: one less. Try to explain to the public that it is only the first line of defense against child sex and that it does not imply anything at all.
1. You would need at least 30 such images;
2. You'd have to somehow convince your target to import all 30 images into their photo gallery;
3. The chances of the images accidentally passing the human review step is infinitesimal;
4. Unlike a bag of drugs which actually do contain drugs, these fake images would be like police finding a bag of sugar with "FAKE COCAINE FOR MANUFACTURING PROBABLE CAUSE" written on it.