Comment by nullc | Hacker News Reader

Comment by nullc | Hacker News Reader

Parent Full thread

> The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges

At a minimum it is known to the state actors generating these databases. Apple itself has directly acknowledged abuse by state actors as a threat vector, so it is entirely inappropriate to rely on obscurity here. (and would be a bad security practice regardless, since presumably other attackers could steal the other algorithims).

> For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM image

This is false. It requires knowing the hash of the target image, which is completely lawful to possess. You don't have to be in possession of the illegal images.

You cannot assume these hashes are impossible to obtain-- because if they were it would mean the whole scheme was pointless (since it would also mean the unlawful images were impossible to obtain). But even if they were impossible to obtain, an attacker could simply create new "apparent" child abuse material submit it to the databases while at the same time creating whatever false matches against it that they wanted.

You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.

> You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.

What I'm saying is that a whole bunch of stars need to align before any practical use for this exploit becomes even remotely plausible. This is slightly more likely if your adversary is a state actor with strong motivations and no issues accessing and handling CSAM. But consider the necessary steps and ask how likely it is:

1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default.

2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database.

3. Acquire the secondary hash algorithm and build a tool that generates hash collisions for that.

4. Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them.

5. Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious.

6. Hope that your target does actually import all thirty images into their iCloud photo gallery.

7. Hope that your hash collsions all work correctly in practice.

8. Hope that Apple's human reviewers screw up.

9. Hope that none of the 30 images cause any of Apple's reviewers to become suspicious, flagging the matter for technical review, causing Apple to blacklist these hashes and rev their hash algorithms in the next version of iOS.

10. Hope that the child safety organisation which Apple reports this to also screws up.

11. Hope that the law enforcement body which this child safety organisation reports this to also screws up.

12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target.

That's a dozen steps, many of which are exceedingly non-trivial. There's probably a dozen more steps I'm missing or unaware of.

Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.

> 1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default.

True for millions of people, also assumes the functionality won't be expanded once initially deployed.

> 2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database.

Authorities claim these databases have millions of hits per year (NCMEC says 21.7 million hits in 2020), so clearly these are not that hard to obtain.

> Acquire the secondary hash algorithm and build a tool that generates hash collisions for that.

Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security.

> Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them.

So. Ordinary, perfectly lawful hardcore pornography-- just minimally selected so that you can't tell how old the talent is. This is indistinguishable from child abuse material especially in review context which has been described (viewing low resolution sobel transformed 'extracts').

But also: At this point when someone is reviewing any of their images the users privacy is already violated. Moreover, a hacker or state actor which has compromised Apple's systems could intercept the content at this point and monitor it regardless of what the review found.

> Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious.

I can't speak to the claimed other hash, but I've demonstrated that for neuralhash you can make images match hashes without looking modified, and certainly without making them look unappealing.

> 6. Hope that your target does actually import all thirty images into their iCloud photo gallery.

well there you have me, because no one who cares about personal privacy or civil rights should ever use an apple device. :)

> 12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target.

Just being accused or questioned is a devastating negative consequence. It's also not too difficult to find examples of people being criminally charged over totally lawful pornography because the prosecutors couldn't bother to use the internet to look up the actresses DOB.

> Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.

That may be true, but it has a different risk reward tradeoff. By your argument swatting would never happen-- it requires a lot of complicated steps, more so than just pulling the trigger on a gun or swinging a baseball bat yourself.

And for what benefit? As the article shows-- these system's are easily evaded. Should our privately owned computing devices be tattling on us based on opaque naughty lists which are cryptographically protected against accountability? -- just to catch some perverts trading widely circulated gross pictures which they're too stupid to fuzz against detection? and not even that-- as you noted in other posts: The pervs will just use other tools. So we degrade everyone's privacy, further establish the precedent that it's morally acceptable for your own private property to conspire against you, and probably don't actually protect any children.

I think we should admit: Especially in light of the news on the same week this was announced about the doctor that was keeping lots of child porn in their iphotos and was caught by another service. The purpose of this scheme isn't to protect children, it isn't to preserve user privacy, it exists to increase and preserve Apple's profit margin by distancing them from creeps and it does so at the expense of the privacy of their users. Apple is converting their customer privacy into a private profit.

ISTM the whole scheme may actually encourage abusing more children: The one reliable way to get sexual pictures of young people that this system cannot catch is to make new images. If this sort of thing were effective it could be expected to enlarge the commercial market for new abuse images by shifting demand from the back catalog of existing material to new material.

I think you're missing the broader point. This isn't about any one step. I know you can make counter-arguments for each individual step in isolation, but in order for this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit.

> True for millions of people

But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here?

> Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security.

With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to.

For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed.

> So. Ordinary, perfectly lawful hardcore pornography--

Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets.

> But also: At this point when someone is reviewing any of their images the users privacy is already violated.

Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.)

> I've demonstrated that for neuralhash you can make images match hashes without looking modified

I haven't seen any examples that didn't look modified. That image of Lenna you posted earlier is quite clearly modified.

> Apple is converting their customer privacy into a private profit.

You couldn't possibly accuse them of doing so surreptitiously.

If there are any consequences of such schemes being implemented, the consequences will have already happened because of similar implementations by Facebook, Google, etc. The only difference with Apple's approach is where certain CPU cycles are occurring. Apple could do all of the hash scanning on cloud servers (like Google) and the functional outcome with respect to your personal liberties would be exactly the same.

> this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit.

>> True for millions of people

> But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here?

Please try to imagine for a moment applying your same style of argument to a typical buffer overflow attack.

Attackers aren't random processes. No CGI is going to get exploited and escalated to root access and website defacement through a random process, sure. Yet the attacks happen.

The fact that the attack only works on 20-40% of people to begin with isn't particularly relevant in modeling the risk.

> With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to. For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed.

The same could have been said about the microsoft PhotoDNA ... they kept it secret to keep "bad guys" from access it. But I have a copy of it now. (also people successfully reproduced it from vague descriptions in academic papers)

> Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets.

I've certainly known people that had their phones full of porn.

> Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.)

Yes they do. They see ("a derivative of") every database match once the user has reached the threshold number of matches.

Edit: On reread I understand what you were arguing there. I misunderstood you at first. I still disagree with you: Images users got from elsewhere are still their private files and are no less so because they didn't make them themselves.

> That image of Lenna you posted earlier is quite clearly modified.

I have a hard time believing that you aren't only saying that because you've seen the original. That image wouldn't strike anyone as unusual.

( nor would https://user-images.githubusercontent.com/858454/130296784-b... )

> You couldn't possibly accuse them of doing so surreptitiously.

I can-- but even if I couldn't, the fact that it's not very surreptitious doesn't make it good.

I think Apple has mislead people about the level of privacy protection provided via cryptowashing, e.g. going on about the private set intersection when it doesn't provide any privacy for your users but just hides the database from the public to prevent accountability. Apple also made misleading claims e.g. about the false positive rate of neuralhash-- claiming that it had "trillion to one" FP rates, when its real FP rate in image collections is higher (e.g. there are colliding imagenet images) and constructed attack images will always FP. The fact that advocates have expended time and effort to dismantle Apple's claims doesn't change that they were misleading to begin with. :)

> The only difference with Apple's approach is where certain CPU cycles are occurring.

To the extent that other parties scanning also applies on person to person messaging that has a reasonable expectation of privacy, I would also hold that the scanning is unethical. But since the service providers own equipment is involved I would also freely admit that the ethically of it is debatable, at least if its disclosed, and that reasonable people could disagree.

In the US this kind of scanning only ends up being lawful at all due to legal inconsistencies. If the communication were not stored it would be an unlawful wiretap (under both federal and many states laws) to scan users private communications. Saving it to a mailspool magically changes the law that applies, even if the behavior visible to the user is identical.

But the distinction that it is the user's on the user's own device eliminates the "it's on the provider's server" excuse. Your own device being programmed against your will to surveil you and report you is just unethical, in my view.

Like many ethical questions there is a boundary. Scanning public communications? Fine. Scanning third party private communications on your own devices? Uhh. Maybe, there be dragons. Your own device scanning your private files and reporting you against your will? Not okay.

Even if the operating behavior is close to the same the principle matters.

You are still trying to skirt round the point that all stars must align for this attack to work. And there are enough stars to make it an absurd proposition. Multiply these likelihoods together and it quickly surpasses the likelihood that the user getting struck by lightning at the exact moment that they win the lottery.

As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I know it's an inappropriate thing to say on HN but claiming that the "image wouldn't strike anyone as unusual" is so far beyond reasonable that I feel myself seriously questioning your sincerity.

> all stars must align for this attack to work.

I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced.

> As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage.

I think it looks like a scan of a crappy smudged 30 year old photograph. Which is also exactly what it is.

FWIW, I thought your comment saying the lena was obviously manipulated was transparently in bad faith, but I held my tongue-- so don't feel too guilty. I would take a sizable wager on that people wouldn't identify it as anything special, and I'm quite surprised to hear that you think differently.

(In one of the threads I posted in it one of the people through it was the barbara image that was changed rather than the lena one...)

> I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced.

I'm not convinced that you're being serious. I rejected that response because it's self-evidently absurd. There were twelve mandatory prerequisites I identified for successful exploitation and the vast majority of them are not under the influence of any attacker, no matter how non-random the hypothetical attacker is.

Do you understand what happens when you're forced to multiply probabilities?

How does this attacker acquire a functioning copy of this secondary hash algorithm, which for all you know is held at least as securely than Apple holds private keys for stuff like iOS cryptographic signing.

How does that attacker change the mind of a person who would never import hard core porn into their phone's photo gallery?

Do you know if Apple's human review requires fooling one reviewer or multiple? I personally have no idea but as someone who has implemented a human review system for critical data, placing such consequential decisions in the hands of a single human is a non-starter. For all you know, the 30 images will be looked at by 30 different human reviewers. How does the attacker ensure that ALL of them will overlook the absence of a PRE-PUBESCENT CHILD in the photograph?

No sir, I do not think you are not being serious. And that's intended as a compliment, because you're clearly a smart person.

Reply on news.ycombinator.com