Hacker News Reader

Comment by tvirosi | Hacker News Reader

Really disgusting idea: I wonder if it's possible for someone to use this as a 'discriminator' in a GAN to configure a generator to recreate the CP this is trying to avoid distributing in the first place.

View on HN

Not really; there's not enough information in the NeuralHashes. You'd get pictures like this,[0] (from [1]) instead.

[0]: https://user-images.githubusercontent.com/1328/129860810-f41...

[1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...

That is assuming that adding plausibility constraints wouldn't fix this issue. I don't know if this is feasible though.

That's not really correct. That's a forcibly created colliding image, that's not the output of the NeuralHash. Also as reported elsewhere, it's absolutely possible to do so.

No, it’s not possible.

If you think there is a credible mechanism, please link to it.

It might have to do with the output possibly being a probability vector as opposed to a binary hash. The whole thing is thus differentiable and optimizable (if a dog image was incorrectly placed in the bad hash bucket it might only be on the border of it while the real CP corresponding to the hash is found at the probabilistic maxima of the hash bucket). Just guessing.

That isn’t correct, nor is it credible.

See: https://www.apple.com/child-safety/pdf/Security_Threat_Model...

Where am I supposed to look in that pdf to understand that it isn't correct or credible? It is certainly true that the model has differentiable and thus optimizable outputs.

Sorry - I posted that link in the wrong place.

Either way, if the claim is that it’s possible to reverse engineer CSAM from the hashes, proof is needed, and nobody has provided even a proof of concept.

The person I responded to was claiming it had been demonstrated. I asked for a link to evidence. You just made a hypothesis about how it might work. That’s not helpful.

Are you trying to stop abuse of children, or enforce a standard that the idea of images of children is bad?

If you’re actually trying to stop abuse, having the computer create fake CP seems like an ideal outcome, since it would avoid the need for abuse of children.

Flooding the market with fakes and then directing consumers of the fakes to whatever mental health resources are available seems like it would fit the claimed problem far better than what apple is currently trying.

This would be bizarre - wouldn't this mean that Apple are essentially shipping illegal images with their OS? (Subject to some as yet unknown decoder)

With the right algorithm you can turn any certain string of bits into a certain other string of bits. So is the image in the data, or is it really in the algorithm?

If the decoder was "trained" on and only works with predictable data, then it might be the algorithm that's illegal, but if a completely new illegal image is created, hashed, fed into the decoder and the decoder produces a valid illegal image, then the illegal data must be in the input, not the algorithm.

This is basically rule 1 of testing neural networks: if the testing data is different from the training data and the results are still correct, your network is "reading" the data correctly and not just memorising a list of known values. I guess this means you'd also need to prove that the decoder doesn't turn most hashes of non-illegal images into illegal images, but if you also did that, you'd have a pretty strong case that the illegal data is in the hash.

Did Apple use the bad images to train the neural network? If yes, I suppose that makes this possibility more realistic.

> Did Apple use the bad images to train the neural network?

NCMEC did, certainly, but I don't think Apple ever got the actual images themselves; just the resultant hashes.

Makes me wonder if there's a possibility of e.g. faces on fbi's most wanted being snuck into the dataset somewhere in the chain.

> if there's a possibility of e.g. faces on fbi's most wanted being snuck into the dataset

Sure, it's possible, but that doesn't seem to have happened in the past decade of PhotoDNA scanning cloud photos to match hashes provided by NCMEC - why would it suddenly start happening now?

You really don't understand the difference in scale distributed sensor netwise between the two different capabilities do you?

Server centric is the primitive that gives you periodic batch. Client resident let's you build up a real-time detection network.

Also, as they say in the financial world: past performance is not indicative of future results. No one would have thought to do so because this step hadn't been done. Now that this step has been done it is an easier to sell prospect. This is how the slippery slope works.

> periodic batch [] real-time detection network

What's the realistic difference here between "my phone scans the photo on upload to iCloud Photos" and "iCloud Photos scans the photo when it's uploaded"?

Latency of upload doesn't come into play here because the scan results are part of the uploaded photo metadata; they're not submitted distinctly according to Apple's technical description.

(And given the threshold needed before you can decrypt any of the tagged photos with the client side system, the server side scanning would be much more "real-time" in this case, no?)

> Sure, it's possible, but that doesn't seem to have happened in the past decade of PhotoDNA scanning cloud photos to match hashes provided by NCMEC

If it's happened, it's unlikely the public would know about it.

Reply on news.ycombinator.com