Proposed illegal image detectors on devices are ‘easily fooled’
imperial.ac.uk
imperial.ac.uk
No one has proposed any of those as 'illegal image detectors'. Apple's NeuralHash may or may not be robust to the same or different perturbations but the cited study provides basically no new information to inform the conversation its press release wants to be a part of.
[0]: https://github.com/facebook/ThreatExchange/blob/main/hashing...
E.g. here is a Lena I modified to match Barbra:
https://user-images.githubusercontent.com/858454/130310372-d...
https://user-images.githubusercontent.com/858454/130310383-9...
Simply making the hash not match in any single bit is so trivial that it's hardly worth trying. If you give me a 360x360 image of your choice though I'll make one for you if you still have any doubts.
Intentional false positives require knowledge of a hash. And even that isn't enough to get a human to scan your image. You would need to acquire a library of A1 CSAM images/hashes. You would need to generate images which match both NeuralHash and the unspecified secondary hash algorithm which runs on Apple's cloud servers. And then you'd need to repeat this trick a sufficient number of times to trigger human review, at which point a few people (probably some minions in a developing country) get to look at the images for 1 second before pressing the "not CSAM" button and having the images sent to a technical team in Cupertino for analysis.
Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery.
Intentional false positives are irrelevant because there are, depending how you count them, between two and six other layers which would need to be thwarted or exploited in order for intentional false positives in NeuralHash to result in any adverse outcome.
So if the purpose of the system is not to catch persons dealing with CP, what is the purpose? Breaking user's privacy?
We have a history of evil intentions masked by fighting against CP, drugs, money laundry and crime in general.
I see no reason to think that general intelligence, technical awareness, utility maximization, impulse control, or risk-reward calculation wouldn’t have similar distributions such that plenty of people would end up in a set prone to share risky material on such channels.
One possibility is that these databases are stuff full of material which isn't actually CSAM. There is no accountability for the content of the databases.
Considering the infinitesimal number of prosecutions compared to the claimed number of hits, we should hope that this explanation is actually the truth...
A system that detects Covid-19 in infected people and doesn't detect it in healthy people, where the errors of both types (false positives and false negatives) are irrelevant, is not only relevant, it's a very well designed system.
So, you're accepting that Apple's approach is useless for anyone who's "determined" but you still seem to be arguing there's some merit in this complex "architecture" of surveillance.
So the idea is that mass surveillance is great even if it's effect is scaring criminals while it actually only catches people at random?
We've arrived at the opposite of the common reprise of law. No longer "It is Better that Ten Guilty Persons Go Free Than that One Innocent Person be Convicted" but "It is better that Ten Innocent Persons Be Dragooned Than that One Guilty Person Go Free"
It's a tough line for them to walk.
Personally I don't see any substantial moral difference to what Apple was planning versus what Google and Facebook already do, aside for the somewhat icky aspect that some initial CPU cycles are occurring on hardware I own.
I'm equally concerned that a purported need to scan for CSAM could be used by legislators as a way to convince the population that the Government needs more encryption backdoors. Personally I'd rather have my photos inconsequentially hashed than have encryption legislatively broken.
Actually, you've made quite a few claims, including some that even ineffective surveillance is good 'cause it makes criminals worried. But anyway...
that I understand why Apple doesn't want CSAM on their servers, while also reserving the ability to implement end to end encryption on photo galleries in future.
Given you've admitted Apple's filtering is ineffective, you have admitted that criminals could put CSAM on Apple servers.
But weird thing is "end-to-end encryption" is entirely useless if everyone is being surveilled before the start and after the end. Sure, Apple wants to be buzzword compliant with privacy and entirely surveil it's users in reality. I suppose I too can see how that would be appealing.
The phrase "all I'm saying is" is an idiom, not a literal assertion.
> Given you've admitted Apple's filtering is ineffective, you have admitted that criminals could put CSAM on Apple servers.
Yes, I did. And I responded to that already. I'm not interested in engaging with "if it's not perfect then it's useless" type of arguments.
Having illegal activity on a major cloud service makes it harder for those services to offer robust encryption without law enforcement pressuring legislators to mandate backdoors.
But in the US it isn't just a backdoor, it's an indirect violation of the user's 4th amendment rights. If the government ordered apple to perform this scanning, the scanning would be unlawful and the result would be inadmissible in court. It is only lawful because Apple performs it voluntarily (or at least that is the pretext maintained in court-- repeatedly Apple staff have maintained to me that the government is requiring them to do this). The government should not get the ability to execute unlawful searches simply because it's able to use soft power to coerce user trusted third parties to perform the searching on their behalf.
The code/network detecting this will be run on the client. So yes, a 'hash' or an equivalent will be known.
You would have to hope that the secondary system is robust enough to not fall victim to the same manipulated image. And it really is on Apple to first prove it isn't.
In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash.
Edit: I am almost entirely wrong, see the comment reply from krisoft. Apologies!
No. You are wrong. They do hash each image and they do upload the hash derived safety voucher always. Here is the Apple technical paper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
There is a fairly complicated cryptographic algorithm which, according to apple, lets them open the safety voucher only if there is a match with the database.
The user devices only have a cryptographicaly blinded version of the hash database which they can’t turn back to real hashes without apple’s private key. This means that the plain neuralhashes are not available to the end users.
They use cryptography to conceal the hash database from the users to guard it against accountability.
Because of this, if they stuff the database with political or ethnically targeted images no one will be able to tell, you just have to trust that they haven't done that.
Knowledge of a matching hash isn't a big barrier for another reason: Matching images are widely available. NCMEC reports that they had over 20 million matching images reported to them last year. It's a very reasonable assumption that some 4channer will helpfully troll the internet for likely illegal images and put up a large database of likely matches.
False positives can also go the other way: Modify illegal images to match existing lawful images which are already in circulation. One could likely take legal hardcore pornography, crop it to make it impossible to identify the ages of the talent, add captions to falsely claim that they are under-age, tweak it to match targeted images, and submit it to the databases.
The encrypted list is stored on the devices for security and performance reasons: otherwise it would be too easy to ship people customized lists with arbitrary images. You can't just rapidly revise the list without undermining the privacy protection here.
Also the space of modifications is gigantic, for any given image there should be millions of differently hash variations which can be reached without major visual damage. (at least for neuralhash the figure is billions)... so you can't just stick all of them in the database.
You could stick widely circulating variations in the database, however... sure.
Are you saying the hash DB can't be updated or that already uploaded photos won't be compared to hash updates?
It can be updated-- at some cost. But if it is very frequently updated then the protection against receiving a targeted list is lost.
To state it another way: Users ask "Why won't the government order apple just gives me a list of the hash of every political image on the internet in order to expose my files?" and the answer is that the database is part of the OS, common to everyone, updated infrequently, so it can't be used in targeted attacks. That protection is lost if the database is frequently updated.
It's also true that already uploaded photos won't be rescanned, but that wasn't the point I was trying to make.
My comment was pointing out that the well known fact that the neuralhash is highly vulnerable to preimage attacks (intentional false positives) also means that is is highly vulnerable to intentional false negatives. Changing the 96-bit hash to a specific value while minimizing the visible image change is a LOT harder than just changing any single bit, but the harder task has been amply demonstrated.
Neuralhash is one component, yes, but it is a limiting component. If a slightly tweaked image has a different NeuralHash it will never be detected by their scheme.
The App Store is presumably full of easy to use apps which can store photo galleries in a separately encrypted vault. This is where people will go after being shooed away from iCloud, not image manglers.
Except that every time they get a positive, the device will upload the image to Apple, where at some point an actual human will have to check it.
Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This is just a slight twist on the common prank of making people Google terms related to CSAM.
Not only will that cause Apple to violate the privacy of innocent people who were just saving cat pictures from Facebook, it will also require a lot more manpower than Apple is probably anticipating to keep up with the amount of false positives that will come in. And that isn't even considering unintentional false positives.
I don't think Apple is planning to start sending out "oopsie we violated your privacy and looked at your memes/family photos" mails to affected people. But it might come to that too once this planned practice is tested against current or future European privacy laws (and such a notification would be the bare minimum in my opinion).
That won't go over well once the first people share such an experience.
1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers.
2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review.
3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced.
4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.
Apple claims without evidence that there is a second secret hash which is run serverside. Perhaps it's just Microsoft PhotoDNA (which isn't secret anymore), perhaps it's just another copy of neuralhash (which might share the same collisions)-- it isn't currently known to the public.
It is, however, known to anyone inside the project at apple and anyone involved in the abuse image databases at a minimum.
I'm of the opinion that any security properties of this supposed second hash should be disregarded in public analysis of the system-- for all we know it's just a "return true". Other people make a reasonable argument that this additional function makes some of the attacks people describe harder. In practice it probably does, but security analysis should concern itself with the worst cases admitted by the known system.
This isn't a reasonable stance, it's just a tactic of poisoning the well such that you can never lose the argument.
I refrained from outright accusing you of bad faith when you claimed that the example image was obviously manipulated, though I confess that was my instant impression. I'm sad that you haven't extended me the same courtesy.
It's considered a bad practice and insecure for systems to keep a cryptographic _algorithm_ secret. In a cryptosystem all that should be secret is the key itself which is small and uniformly randomly generated and by construction cannot contain any weaknesses that could be uncovered by inspection (as it's just uniformly selected out of the space of possible keys).
This doesn't at all apply to a perceptual hashing system: the system will certainly contain weaknesses which could be found by chance or by limited black box interrogation, but it could only be cheaply reviewed for flaws with direct access to it.
The fact that secure cryptosystems by construction protect the keys from leakage through usage is part of the reason that it's okay to stuff all the security dependency into the key.
If you used HMAC with an unknown key and allowed me oracle access to see if inputs I sent in matched, I couldn't learn anything about your hmac key (unless the underlying hash was broken). But if we didn't know the weights of neuralhash and I got match oracle access to it and access to a few matching images (maybe 1 per bit of hash), I could likely the entire set of weights just through interrogating an oracle that gives a pass fail. (Also the papers that successfully design adversarial inputs for unknown networks, simply by designing adversarial inputs for a set of entirely different networks)
Moreover, this second perceptual hashing system requires that Apple share the system with third parties-- they must share it with e.g. NCMEC and the agencies of other governments because it is a felony for apple to knowingly handle the unlawful images themselves so they cannot produce the database themselves, it has to be produced by NCMEC that has the special statutory carveout. By contrast, in a standard public key encryption or authentication scheme your private key is shared with no one ever. That makes it much easier to keep secure (though keys often do get leaked and stolen).
Finally, if I steal your pgp key or whatever, it effects a single use or a single user. Most cryptosystems use as many keys as reasonably possible so the loss of one has the limited impact-- and they can be replaced with another uniformly generated random key if there is a concern for a key's security. But there is only one second image hash, they likely cost millions to develop, so the leakage of it would be much more damaging to the system compared to leaking a key.
I'm perfectly willing say that we have a reasonable disagreement on value of the 'secret' additional hashing scheme. The role of obscurity in security systems has been debated for decades in forums far beyond here-- and we're not likely to add more to it in this discussion.
But please don't accuse me of poisoning the well because we have opposing perspectives on this broad subject. I think I've adequately argued why your comparison point isn't a great comparison. We might not agree but we should agree that neither of us is taking a disingenuous position.
Security by obscurity in 2021?
The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.
That's good, because they weren't arguing that NeuralHash is infallible, but that its fallibility does not lead to the consequences outlined by the GP.
> The fact is that a 'first line of defense' that is trivial to fool is useless.
Useless to you, but not to apple. It reduces the set of pictures to process from "everything" to "those identified by the algorithm as designed, plus artificial false positives". It's impossible for the second set to be larger than the first, so it's by definition not useless.
This is why it's strange how people are focusing so hard on the weaknesses of NeuralHash when what we should care about is how well the system as a whole works.
I presume Apple didn't break the law and morals to construct the hashes.
I am being facetious, I know, but my point is that humans' morals and ethics are incredibly widely diverse ― especially if we're talking about the "effective" instead of "proclaimed" ethics (i.e., the values people actually follow instead of values they believe/claim to follow).
This might actually be brilliant and we might end up thanking them for it. It might pretty much destroy this filter.
> This is just a slight twist on the common prank of making people Google terms related to CSAM.
And here is the problem: there is one failure mode of this and that is if a bunch of ordinary people get jailed for innocent memes before the public gets whiff of whats happening.
So if they do they should probably be targeted as a waterhole "attack" at Apple execs and pro censorship politicians.
Apple employees reading this: think very closely before downloading wallpapers the next few years.
That's a very optimistic view.
In reality, they need to automate something while they never get any feedback whatsoever if their efforts are successful, let alone worthwhile.
Crooks that attack Paypal at least know when they succeed - they see the money. In this case, not only they won't see the money, they won't know if what they're doing is "overwhelming" the system or not.
Complaints about the proposal seem to overly focus on the reliability of singular aspects of the automated detection system when this has already been planned for - additionally those comments seem entirely unaware of the threshold technology in play.
Cops plant illegal drugs on suspects all the time. Why not this?
BANG: one less. Try to explain to the public that it is only the first line of defense against child sex and that it does not imply anything at all.
1. You would need at least 30 such images;
2. You'd have to somehow convince your target to import all 30 images into their photo gallery;
3. The chances of the images accidentally passing the human review step is infinitesimal;
4. Unlike a bag of drugs which actually do contain drugs, these fake images would be like police finding a bag of sugar with "FAKE COCAINE FOR MANUFACTURING PROBABLE CAUSE" written on it.
The insidious thing is that this can used as a pretext to make the filters flag more images since this would seem to make it "harder to hide illegal images", until Apple just personally scans everything.
That Pandora's box is locked pretty tightly.
If you work for a company advancing this technology you are evil and should rethink your life choices.
(Your second paragraph allows for no context or nuance and is just completely unnecessary.)
A person's expectations and rights relating to a discussion on a public forum are nothing like those relating to a device you own.
GP's "pretty hot take" is correct.
I don’t think my “rights” related to accessing my OS software is much different than my rights related to accessing discussion forum software.
Seems pretty apples to apples to me.
(I don’t like image scanning, and I don’t like wasted brain cycles, if that matters)
Before buying the device? Sure, if you want. You would be aware that those wasted cycles and wasted network traffic are part of the price.
But what about the rest of us who aren't like you, who bought the device without knowing before hand that this theft would take place?
All words that have meaning that becomes diluted in public perception, and lead to things sort of becoming a "black box", where we don't really know how the models actually do what they do.
Seems history it repeating itself with "Artificial intelligence" with all things done by a computer...
I'd wager that project managers working on these teams also don't have the understanding (let alone intuition) to judge whether these risks are present, and so will continue fighting for it. The wager is based on the presumption that someone who did understand these risks would pivot or otherwise not allow this initiative to continue.
PMs fight so hard because from their position their job is at risk if the project takes a nosedive. But they don't know what they're advocating for a lot of the time.
I know quite a few folks like this in FAANG and adjacent spaces.
“These images have been hashed, so that they look different to detection algorithms but nearly identical to us”
Or did I miss the sarcasm tag?
Talking about how trivial it is on a technical level is, in my opinion, an oversimplification. For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM images. These are images of sexually explicit material where pre-pubescent children are actively involved.
Following introspection of my own personal moral compass, it's clear to me that being knowingly in proximity to such material would not be "easy" for me.
do you really think there won't be a large list of CSAM hashes on torrents if you go look for it?
But maybe you could. Let's say hypothetically, such a torrent does exist. Apple could just rev their hash algorithm in the next version of iOS, immediately rendering those lists useless.
At a minimum it is known to the state actors generating these databases. Apple itself has directly acknowledged abuse by state actors as a threat vector, so it is entirely inappropriate to rely on obscurity here. (and would be a bad security practice regardless, since presumably other attackers could steal the other algorithims).
> For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM image
This is false. It requires knowing the hash of the target image, which is completely lawful to possess. You don't have to be in possession of the illegal images.
You cannot assume these hashes are impossible to obtain-- because if they were it would mean the whole scheme was pointless (since it would also mean the unlawful images were impossible to obtain). But even if they were impossible to obtain, an attacker could simply create new "apparent" child abuse material submit it to the databases while at the same time creating whatever false matches against it that they wanted.
You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.
What I'm saying is that a whole bunch of stars need to align before any practical use for this exploit becomes even remotely plausible. This is slightly more likely if your adversary is a state actor with strong motivations and no issues accessing and handling CSAM. But consider the necessary steps and ask how likely it is:
1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default.
2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database.
3. Acquire the secondary hash algorithm and build a tool that generates hash collisions for that.
4. Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them.
5. Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious.
6. Hope that your target does actually import all thirty images into their iCloud photo gallery.
7. Hope that your hash collsions all work correctly in practice.
8. Hope that Apple's human reviewers screw up.
9. Hope that none of the 30 images cause any of Apple's reviewers to become suspicious, flagging the matter for technical review, causing Apple to blacklist these hashes and rev their hash algorithms in the next version of iOS.
10. Hope that the child safety organisation which Apple reports this to also screws up.
11. Hope that the law enforcement body which this child safety organisation reports this to also screws up.
12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target.
That's a dozen steps, many of which are exceedingly non-trivial. There's probably a dozen more steps I'm missing or unaware of.
Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.
True for millions of people, also assumes the functionality won't be expanded once initially deployed.
> 2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database.
Authorities claim these databases have millions of hits per year (NCMEC says 21.7 million hits in 2020), so clearly these are not that hard to obtain.
> Acquire the secondary hash algorithm and build a tool that generates hash collisions for that.
Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security.
> Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them.
So. Ordinary, perfectly lawful hardcore pornography-- just minimally selected so that you can't tell how old the talent is. This is indistinguishable from child abuse material especially in review context which has been described (viewing low resolution sobel transformed 'extracts').
But also: At this point when someone is reviewing any of their images the users privacy is already violated. Moreover, a hacker or state actor which has compromised Apple's systems could intercept the content at this point and monitor it regardless of what the review found.
> Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious.
I can't speak to the claimed other hash, but I've demonstrated that for neuralhash you can make images match hashes without looking modified, and certainly without making them look unappealing.
> 6. Hope that your target does actually import all thirty images into their iCloud photo gallery.
well there you have me, because no one who cares about personal privacy or civil rights should ever use an apple device. :)
> 12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target.
Just being accused or questioned is a devastating negative consequence. It's also not too difficult to find examples of people being criminally charged over totally lawful pornography because the prosecutors couldn't bother to use the internet to look up the actresses DOB.
> Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.
That may be true, but it has a different risk reward tradeoff. By your argument swatting would never happen-- it requires a lot of complicated steps, more so than just pulling the trigger on a gun or swinging a baseball bat yourself.
And for what benefit? As the article shows-- these system's are easily evaded. Should our privately owned computing devices be tattling on us based on opaque naughty lists which are cryptographically protected against accountability? -- just to catch some perverts trading widely circulated gross pictures which they're too stupid to fuzz against detection? and not even that-- as you noted in other posts: The pervs will just use other tools. So we degrade everyone's privacy, further establish the precedent that it's morally acceptable for your own private property to conspire against you, and probably don't actually protect any children.
I think we should admit: Especially in light of the news on the same week this was announced about the doctor that was keeping lots of child porn in their iphotos and was caught by another service. The purpose of this scheme isn't to protect children, it isn't to preserve user privacy, it exists to increase and preserve Apple's profit margin by distancing them from creeps and it does so at the expense of the privacy of their users. Apple is converting their customer privacy into a private profit.
ISTM the whole scheme may actually encourage abusing more children: The one reliable way to get sexual pictures of young people that this system cannot catch is to make new images. If this sort of thing were effective it could be expected to enlarge the commercial market for new abuse images by shifting demand from the back catalog of existing material to new material.
> True for millions of people
But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here?
> Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security.
With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to.
For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed.
> So. Ordinary, perfectly lawful hardcore pornography--
Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets.
> But also: At this point when someone is reviewing any of their images the users privacy is already violated.
Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.)
> I've demonstrated that for neuralhash you can make images match hashes without looking modified
I haven't seen any examples that didn't look modified. That image of Lenna you posted earlier is quite clearly modified.
> Apple is converting their customer privacy into a private profit.
You couldn't possibly accuse them of doing so surreptitiously.
If there are any consequences of such schemes being implemented, the consequences will have already happened because of similar implementations by Facebook, Google, etc. The only difference with Apple's approach is where certain CPU cycles are occurring. Apple could do all of the hash scanning on cloud servers (like Google) and the functional outcome with respect to your personal liberties would be exactly the same.
>> True for millions of people
> But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here?
Please try to imagine for a moment applying your same style of argument to a typical buffer overflow attack.
Attackers aren't random processes. No CGI is going to get exploited and escalated to root access and website defacement through a random process, sure. Yet the attacks happen.
The fact that the attack only works on 20-40% of people to begin with isn't particularly relevant in modeling the risk.
> With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to. For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed.
The same could have been said about the microsoft PhotoDNA ... they kept it secret to keep "bad guys" from access it. But I have a copy of it now. (also people successfully reproduced it from vague descriptions in academic papers)
> Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets.
I've certainly known people that had their phones full of porn.
> Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.)
Yes they do. They see ("a derivative of") every database match once the user has reached the threshold number of matches.
Edit: On reread I understand what you were arguing there. I misunderstood you at first. I still disagree with you: Images users got from elsewhere are still their private files and are no less so because they didn't make them themselves.
> That image of Lenna you posted earlier is quite clearly modified.
I have a hard time believing that you aren't only saying that because you've seen the original. That image wouldn't strike anyone as unusual.
( nor would https://user-images.githubusercontent.com/858454/130296784-b... )
> You couldn't possibly accuse them of doing so surreptitiously.
I can-- but even if I couldn't, the fact that it's not very surreptitious doesn't make it good.
I think Apple has mislead people about the level of privacy protection provided via cryptowashing, e.g. going on about the private set intersection when it doesn't provide any privacy for your users but just hides the database from the public to prevent accountability. Apple also made misleading claims e.g. about the false positive rate of neuralhash-- claiming that it had "trillion to one" FP rates, when its real FP rate in image collections is higher (e.g. there are colliding imagenet images) and constructed attack images will always FP. The fact that advocates have expended time and effort to dismantle Apple's claims doesn't change that they were misleading to begin with. :)
> The only difference with Apple's approach is where certain CPU cycles are occurring.
To the extent that other parties scanning also applies on person to person messaging that has a reasonable expectation of privacy, I would also hold that the scanning is unethical. But since the service providers own equipment is involved I would also freely admit that the ethically of it is debatable, at least if its disclosed, and that reasonable people could disagree.
In the US this kind of scanning only ends up being lawful at all due to legal inconsistencies. If the communication were not stored it would be an unlawful wiretap (under both federal and many states laws) to scan users private communications. Saving it to a mailspool magically changes the law that applies, even if the behavior visible to the user is identical.
But the distinction that it is the user's on the user's own device eliminates the "it's on the provider's server" excuse. Your own device being programmed against your will to surveil you and report you is just unethical, in my view.
Like many ethical questions there is a boundary. Scanning public communications? Fine. Scanning third party private communications on your own devices? Uhh. Maybe, there be dragons. Your own device scanning your private files and reporting you against your will? Not okay.
Even if the operating behavior is close to the same the principle matters.
As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I know it's an inappropriate thing to say on HN but claiming that the "image wouldn't strike anyone as unusual" is so far beyond reasonable that I feel myself seriously questioning your sincerity.
I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced.
> As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage.
I think it looks like a scan of a crappy smudged 30 year old photograph. Which is also exactly what it is.
FWIW, I thought your comment saying the lena was obviously manipulated was transparently in bad faith, but I held my tongue-- so don't feel too guilty. I would take a sizable wager on that people wouldn't identify it as anything special, and I'm quite surprised to hear that you think differently.
(In one of the threads I posted in it one of the people through it was the barbara image that was changed rather than the lena one...)
I'm not convinced that you're being serious. I rejected that response because it's self-evidently absurd. There were twelve mandatory prerequisites I identified for successful exploitation and the vast majority of them are not under the influence of any attacker, no matter how non-random the hypothetical attacker is.
Do you understand what happens when you're forced to multiply probabilities?
How does this attacker acquire a functioning copy of this secondary hash algorithm, which for all you know is held at least as securely than Apple holds private keys for stuff like iOS cryptographic signing.
How does that attacker change the mind of a person who would never import hard core porn into their phone's photo gallery?
Do you know if Apple's human review requires fooling one reviewer or multiple? I personally have no idea but as someone who has implemented a human review system for critical data, placing such consequential decisions in the hands of a single human is a non-starter. For all you know, the 30 images will be looked at by 30 different human reviewers. How does the attacker ensure that ALL of them will overlook the absence of a PRE-PUBESCENT CHILD in the photograph?
No sir, I do not think you are not being serious. And that's intended as a compliment, because you're clearly a smart person.
https://www.theverge.com/2015/7/24/9032179/chrysler-announce...
I suspect that individuals working with sensitive information in other countries do not use iOS or official Android when they are at work.
If US spies on allies it would probably spy 100x more the non alied states. https://www.reuters.com/world/europe/us-security-agency-spie...
Is this a bug or a feature? Yes.
Meaning the image detectors would be illegal in most countries if actually implemented?
Would this also be good for copyright detecting?
Whoever said it would be a magic bullet?
> The researchers say this highlights just how easily people with illegal material could fool the surveillance.
Ha yes, all you need to do is go to a university computing department and ask them to research algorithms to fool the scanner, and then turn it into an easy-to-use app.
Then you chance it with some real CP images. It might work! Or not.
Interesting but I don't think anyone at Apple will be shocked by this.
Apple has a vested interest in preventing iCloud from becoming "That image storage place for pedophiles."
Also, I'm sure that the FBI is probably pretty okay with "we only catch the people who ever, in a single instance, forget to filter their images through this program that fools the filters" since that's probably like 99% of all people.
It's worth stopping to consider that while Apple's plans were a hot topic for the the civil liberties crowd, that's probably piss in the wind compared to how hard the announcement landed among underground CSAM communities across the globe. I dare say that even the most technically inept CSAM aficionados will be acutely aware of this looming threat and will know to keep that shit far away from any of their Apple devices.
(It would be especially ironic if there's now more awareness in those communities of Apple's semi-aborted CSAM scanning than of Google or Facebook's CSAM scanning, which has been operational for many years.)
When dealing with demagogues it is time to go on the offense - there is a reason why McCarthy was damned by the line "Have you no decency?!"
Also, for the more cynical, here's a headline from 2016 Washington Post:
“Joe McCarthy was brought down by attacks on his decency. Trump will lose the same way.”
Then why bother with the false-positive-risk inducing neuralhash to being with? Compute the sha256 sum of the decompressed image pixels and use that... at that point a false positive is a successful attack on sha256.
The only downside is that trivial image modification will make the hash not match, but that's also true for the perceptual hashes... just with a slightly more complicated definition of trivial. It will be equally effective against the idiot-pervert -- arguably more effective because there would be a lot less reason for people oppose the deployment of such a scheme.
Using a cryptographic hash would make the software vastly simpler, vastly faster, and effectively eliminate a whole host of privacy attacks.
They could then go further and stop using the cryptographic obfuscation that protects apple against accountability for the content of the database-- just make the database the cleartext sha256 hashes. Doing this would substantially address a whole host of other privacy concerns and would make the system radically simpler and less cpu hungry.
They can't do that with the neuralhash because of how trivial it is to modify images to match target hashes... but after switching to a cryptographic hash they certainly could. (Well, could except the database providers know their database contains a lot of lawful content and are extremely aggressive at preventing the lists of hashes from getting out).
Well, the difference between the two version of "trivial" is the difference between "incidental" and "purposefully".
Re-compressing, resizing or cropping the image changes the SHA256, but (with high probability) not the NeuralHash. Those operations happen all the time in the normal course of image distribution. Unlike "running through a tool designed to alter the NeuralHash value".
In other words, your suggestion trades false positive rate for false negative rate, under the argument that any nonzero false negative rate is effectively the same. This argument is not convincing, particularly when your suggestion would reduce the true positive rate from (a hypothetical) 99% to a much lower value.
This would be robust against a significant percentage of recompression pipelines, and yet would still maintain a clear guarantee against false positives: Unless SHA256 is successfully attacked, no one can find two images with the same hash unless their 360px/6-bit-per-color downsample is identical. (And small crops-- are they really that common?-- could be handled by adding them to the database. For significant crops the neuralhash won't be preserved in any case.)
I'm aware that attackers can make contrived images that look rather different when downsampling. But that is still a much much stronger promise than neuralhash could make since the downsample hack requires obviously contrived false positive images (unlike neuralhash false positive images).
Essentially neuralhash (and similar approaches) is the solution you get when you weigh maximally in favor of false positives to avoid false negatives. SHA256 the pixels (or the whole image files) is the solution you get when you favor false negatives to avoid false positives. You can hit many points in the middle.
Given that no scheme can be robust against intentional false negatives and that apple is happy to allow 30 hits before treating it as a match, I'd argue that the apple scheme (or similar) is just an objectively bad choice.
Let's assume image compression randomly flips the last bit of every pixel (this is a crude approximation but it should do).
Let's further assume your original image was 1k x 1k and was shrunk by a factor of 3, so each downscaled pixel is the average of 9 original ones.
Let's then assume that all nine pixels have to "flip" in a specific way to change the downscaled one, so a 1/2^9 chance.
The odds of this bit flip propagating past the last two bits you dropped are 1/4 (dropping bits is not very effective at combating random behavior!).
That means there is a 1/2^11 chance that adding compression to the pipeline changes any given downsampled channel-pixel in the 360px/6-bit-per-color hash input.
The SHA256 stays the same if this does not happen in any of the 360 x 360 x 3 channel-pixels. The odds of that are (1 - 1/2^11)^(360*360*3) = 3.4e-83. You could drop two more bits and the odds would still be 2.4e-21 (each dropped bit amounts to taking the square root).
In effect, your suggested scheme would simply never assign the same hash with any amount of compression happening.
Fundamentally, you still have way too many pixels that might spuriously change from tiny perturbations of the input image. Reducing resolution further is actually much more effective than dropping bits, because it both increases the averaging effect and reduces how many pixels have a chance to flip.
1. It's only done on cloud photos. That person would have to manually save each photo to their photos library (with iCloud photos turned on) for the scanning to occur.
2. After reaching a threshold # of photos automatically flagged, a human reviewer has to confirm that the photos are illegal before any action is taken.
So just texting someone even real CP won't do anything.
1. The account is for a child under the age of 13
2. The account was configured with family sharing
3. The feature was enabled by a parent
Results of these scans would remain entirely local to the device and—with the consent of the child—also sent to the parent. But I think this last feature has been dropped.
One feature does blind CSAM detection on iCloud photos. Any photos that are being added to iCloud in the USA have a safety voucher calculated on-device and attached to the upload. The device doesn’t know if there’s a match or not. When the uploads reach Apple’s servers, if there are enough matches, Apple has enough information to decrypt the safety vouchers. The photos are then additionally compared with a second, secret hashing algorithm. If those also match, a human reviewer checks a low-res thumbnail to confirm. If they also confirm, it gets reported. So the initial calculation of the safety voucher is done on-device, but the matching can only be completed on the server, and nothing gets reported unless multiple other checks also match.
The other feature uses machine learning to determine whether or not a photo attached to a message contains nudity. If such an image is received by a child’s iPhone, it’s blurred out and they are told that if they choose to view it, their parents will be told about it. The entirety of this scanning happens on-device.
The reviewer actually only confirms that the-photos-as-modified-by-an-undisclosed-transformation are illegal, and this strategic ambiguity is presumably designed to maintain plausible deniability that what Apple subjects its reviewers to is a) not psychologically damaging, and b) not illegal to possess.
Unfortunately this is "security by obscurity", so we have to assume it is vulnerable to image scaling attacks[0].
Unless Apple document their human review process, and allow an independent audit of the technology (to provide a strong mathematical guarantee that it is not susceptible to such image-smuggling attacks) and independent verification that the published process is the one they actually use, then there is an unacceptably high chance that a human reviewer will trigger a legal process against users for files they couldn't tell contained illegal data.
[0] https://embracethered.com/blog/posts/2020/husky-ai-image-res...