Hacker News Reader

Comment by scotty79 | Hacker News Reader

Why are exact collisions interesting? They are not intended to be compared exactly.

This algorithm doesn't even give exact matches for the same image on different hardware.

https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX

Note: Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors.

View on HN

The hash is 96 bits long. When hashing 1 billion pictures, that gives a collision probability of 6e-12. If it were uniformly distributed. There's no way people have hashed billions of images already. It just shows that it's pretty probably there will be collisions, and on visual inspection, it looks as if the collisions will happen on visually similar images. So if there's a naked baby pic in the CSAM database, quite a few of you 100s of child pictures can be flagged.

Clearly this is not a cryptographic hash, and hence it's known hashes are not uniformly distributed.

Apple explained in their technical summary [0] that they'll only consider this an offence if a certain number of hashes match. They estimated the likelihood of false positives there (they don't explain which dataset was used, but it was non-CSAM naturally) is 1 out of a trillion [1]

In the very unlikely event where that 1 in a trillion occurrence happens, they have manual operators to check each of these photos. They also have a private model (unavailable to the public) to double-check these perceptual hashes which also used before alerting authorities.

[0] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... [1] https://www.zdnet.com/article/apple-to-tune-csam-system-to-k...

> they have manual operators to check each of these photos

For now. But what will happen when there are thousands of false positives per day? Will they increase the staff? Or will they add another algorithmic layer? Or just up the threshold a bit? There's no guarantee. The only thing that's certain is that the NeuralHash doesn't inspire confidence.

Regardless of what Apple does, law enforcement must always manually review suspected CSAM before requesting a warrant based on it. So the idea that you could SWAT someone with some hash collisions on innocent images is just not possible.

At most you could maybe temporarily lock someone’s iCloud account. But again, the collisions would need to be multiple and all look like CSAM at reduced resolution.

In general, it seems not correct to think about NeuralHash like SHA or RSA. It’s not a cryptographic system and collisions are not a one-step endgame.

> Regardless of what Apple does, law enforcement must always manually review suspected CSAM before requesting a warrant based on it. So the idea that you could SWAT someone with some hash collisions on innocent images is just not possible.

A quick search found four clear cases where law enforcement has favored technological false positives over evidence:

Ousmane Bah: https://www.businessinsider.com/teen-sues-apple-1-billion-fa...

Robert Williams: https://www.cbsnews.com/news/facial-recognition-60-minutes-2...

Nijeer Parks: https://www.cnn.com/2021/04/29/tech/nijeer-parks-facial-reco...

Michael Oliver: https://www.dailydot.com/debug/detroit-facial-recognition-wr...

The one on Ousmane Bah really frustrates me- not only was he on a date at prom during the theft, he was in another state! "Nothing to hide" does not mean "nothing to fear" and allegations (even false ones) of possessing CSAM will ruin lives.

At the end of the day, what it really comes down to is trust; personally, I do not have enough faith in due process to not ruin innocent people. But I'm just some guy online, I will readily admit I don't know anything about anything.

Afiu, Apple's NeuralHash uses exact collisions when they do their Private Set Intersection.

The main advantage of using exact collision is that you can then blind the perceptual hash with a cryptographic hash and avoid any leak of information. (Taking for example sha256 of this perceptual hash won't allow any attacker to get any information on the features from the hash, but if the perceptual hash are the same then the input of the sha256 is the same and therefore the output of the sha256 is the same).

This is important because it alleviates the risks of an eventual leaking the database as Apple never touched and compared sensitive content but only cryptographic hashes of the perceptual hashes.

Some other system like PhotoDNA, rely on a euclidean distance between features being less than a threshold to register a match, which allows to quantify how far the image is from CSAM, but mean that the hash leak some information about the original content.

But they can't use exact matches because their algorithm doesn't create same hash for same image on their various platforms due to hardware differences in floating point arithmetic on their different devices. Unless they have on their servers floating point arithmetic emulators that can calculate exact hash for each of their different devices for each offending image then they can't only match exactly.

Per ATP: Apple will compare hashes of local photos with a national registry of child pornography photos. Once a certain (unknown) threshold is reached, let's say 20 hits, some kind of escalation occurs, with some kind of manual (human) review steps.

Accidental Tech Podcast - A Storm of Asterisks https://atp.fm/443

I haven't listened to the follow up episode yet.

I still have zero opinion on this photo scanning kerfuffle. I just don't know enough. Of all the "hot takes" on this issue, ATP's has been the most comprehensive. So appreciated.

Thanks for the link. Daring Fireball’s analysis is the most helpful take I’ve seen on the topic so far:

https://daringfireball.net/2021/08/apple_child_safety_initia...

I'm interested in the details of the manual review. Does Apple have access to the database of original images and will they use it compare? If not, I can imagine a scenario where a photo of a naked child is flagged as matching the database, the human reviewer sees that it is in fact a naked child and assumes this must mean the image has been legally determined to be child pornography. The case gets referred to authorities and the innocent victim's life is upended for potentially years until the case plays out.

Law enforcement must manually review all suspected CSAM before seeking a warrant based on it. There is a whole process there, beyond whatever Apple has implemented, before a prosecution begins.

Understand that matching a file in the NCMEC database is not itself a crime. The whole CSAM-detecting ecosystem is just a tool for surfacing potential crimes. Having a few pics of your own child naked is not illegal and it’s pretty easy for law enforcement to figure out if that’s the case.

>Law enforcement must manually review all suspected CSAM before seeking a warrant based on it.

Is this a legal statute or simply convention due to the ways things have historically worked (i.e. pre-hash matching at scale)? If warrants are granted based on probable cause, it seems easy to convince a judge that a hash match is sufficiently unlikely that it would exceed the threshold for probable cause. In the context of cryptographic hashes, this is accurate. But if law enforcement doesn't distinguish between cryptographic and perceptual hashes, then there is the real possibility for cases opened and warrants issued unjustifiably.

Sure, matching a hash isn't a crime and you will eventually be exonerated. But as they say, you can beat the rap but you can't beat the ride.

> Sure, matching a hash isn't a crime and you will eventually be exonerated.

s/will/might/

So there's a national database of all child pornography that Apple has access to? How is that access legal?

That is a good point; has Apple stated how many bits two images’ NeuralHashes can differ by and still be considered a “match” by their system?

> Why are exact collisions interesting?

1. What does “exact” mean to you in this context?

2. What else is more interesting about a hashing algorithm used to identify things, other than its collision rate?

Ad.1 Same value of each of 96 bits of the hash.

Ad.2 If hashes are to be matched approximately, not exactly, for example will be considered a match if they differ in less than 3 bits out of 96 then the most interesting thing should be how many collisions you can find if you compare them like that.

That explanation doesn’t make sense to me. Standard (IEEE) floating-point math is deterministic; it shouldn’t depend on the hardware.

The processors that implement hardware-accelerated neural nets are not necessarily using standard IEEE floating-point math.

> They are not intended to be compared exactly.

Apple's private set intersection which leaks the keys to decrypt the images coniditional a neuralhash match requires an exact match.

They probably didn't realize they got different results on different toolchains/devices, since they target a mono-culture and the whole subsystem shows fairly little careful thought went into it. They could easily make an exact integerized version which would be consistent.

It would still be broken. :)

... does it have to prove Apple wrong about something to be interesting?

If you can get exact collisions, this can be gamed. For example, suppose there are two rival gangsters. One wants to set the police on his rival. He knows that a certain (innocuous) image is on his rival's phone. So he pays someone to generate a fake child-porn image with the same neuralhash, and ensure that it gets into the child porn DB. Then, apple reports the rival to the police, and they come and investigate him. OF course, they may notice that the image isn't the right one, but by that time they may have found other incriminating evidence.

Not sympathetic to a rival gangster? Ok lets find an innocent victim: not a rival criminal, but an innocent witness who our protag wants to intimidate. Gangster wants to intimidate the witness, but can't get at them, so cooks up a scheme to convince the witness that the police are in his pocket. Exactly as above, causing the police to investigate the witnesses phone.

Another one might be, a certain government wants to identify opposition groups using images associated with them . Apple is not keen to be associated with that, but the government can simply generate fake child-porn (remember, programmatically generated CP is just as illegal) for each image of interest.

I think before a criminal investigation, or any investigation at all is pursued, a human verifying the images would dismiss the false positive.

I would think surreptitiously placing actual child porn on a rival's phone/computer would be much, much more effective.

Cybercriminals could likely do all this remotely. Phish for apple account login, upload images. Done.

you think the US government would respect the law before initiation of actions to take down a criminal? they rarely do that when dealing with people that dont have criminal backgrounds

If you really believe you live in a country with a completely corrupt legal system, then this stuff is the least of your worries.

Okay. Then why are you worried about this new system, given that the US govt can just throw you in prison at a whim?

So you think an underpaid worker in India is interested in the upkeeping of innocence of a person they don't know in the US? They'll just flag the images as fast as possible, they are probably paid by pictures-reviewed.

"Amazon has people transcribing audio in Costa Rica, India and Romania." according to Engadget. So it would be safe to assume Apple does something along those lines, I can't remember where Siri transcripts have been sent when there was the Alexa controversy.

From there it's an entry with law enforcement and you need to find a way to convince them that the images are your children.

And if data isn't cleared correctly and you have another run in with law enforcement, there will always be this picture-stuff.

> a human verifying the images would dismiss the false positive

How is a human supposed to distinguish that a visual derivative (a low res sobel filtered image, presumably) of ordinary, lawful, adult pornography isn't child porn when the system has already identified it as such?

I agree that using real child porn is an attack too, but at least in that case you could say the system was doing as designed (even though what its doing shouldn't be something that we want) ... but it's not even guaranteed to do as designed.

Are you saying humans cannot visually inspect the actual image that the flag is set against? If not, then how on earth do you prosecute someone if you can't demonstrate they had the actual image, and not just the derivation used to flag it?

The way I see it working is Apple scans a ton of shit, some of it shows up as possible child porn, human intervenes and looks at the source images, if there is indeed child porn they report to the police.

Apple's review has no access to the matched images in the database, just a 'derivative' of your image. Hopefully the prosecutors will check against the real images. But by that point your privacy has been damaged and your reputation might be trashed.

If your enemy can get an image on your phone and in the “child porn DB”, I think they can easily get you in trouble without Apple’s help.

they can either just send the police an anonymous message or set up a child porn web site and have it ‘accidentally’ leak its password database, and make sure your email address is in it.

The point is that they don't need to get an image on your phone. They can just choose one they know is there.

Reply on news.ycombinator.com