Comment by joe_the_user | Hacker News Reader

Comment by joe_the_user | Hacker News Reader

Parent Full thread

Just from memory when the scheme came up in earlier discussion.

The system is essentially scanning for the signature for some known set of images of abuse so that it aims to capture abusers who would naively keep just these images on their machines. (It can't determine if a new image is abusive, notably).

It's conceivable some number of (foolish and abusive) people will be caught this way and those favoring a long dragnet for this stuff will be happy. But this opens the possibility that a hacker could upload an image to an innocent person's computer and get that person arrested. Those favoring the long dragnet will naturally say the ends justify the means and you can't make an omelet without cracking a few eggs. Oh, "think of the children".

Edit: Also worth adding that once a company is scanning user content to try to decide if the user is bad, it makes it that much easier to scan all kind of content in all kind of ways for all kinds of reasons. "for the good", naturally.

That was the CSAM thing that they have announced they gave up on.

This is totally different.

Are you sure? Publicly yeah, but the same technology can easily be used for the same purpose.

As pointed out in a sibling comment, the result set is also encrypted, so matches with abuse images, even if there are some in Apple's POI database, can't be used to implement the scheme as you suggest.

Only in a very broad sense that they use HE to prevent the server from seeing what happened.

The CSAM tech detected matches against particular photos captured by law enforcement, and provided external evidence of the match (e.g. enough positive matches reconstructed a private key). It was not meant to do topical matches (e.g. arbitrary child in a bathtub), and had some protections to make it significantly harder to manufacture false positives, e.g. noise manipulated in kitten photos to cause them to meet the threshold to match some known image in the dataset.

This gives a statistical likelihood of matching a cropped image of a landmark-like object against known landmarks, based on sets of photos of each landmark (like "this is probably the Eiffel Tower"), and that likelihood is only able to be seen by the phone. There's also significantly less risk about around abuse prevention (someone making a kitten photo come up as 'The Great Wall of China')

>The system is essentially scanning for the signature for some known set of images of abuse

Huh? The system is scanning for landmarks, not images of abuse.

>people will be caught this way

Due to the homomorphic encryption, I don't think Apple even knows whether the image matches a landmark in Apple's server database or not. So even if Apple put some images of abuse into its server database (which Apple claims only contains pictures of landmarks), I don't think Apple would know whether there was a match or not.

Does Apple explicitly say that?

Or only that they don’t know which landmark it matched?

Fundamentally, vector search like this doesn't have a concept of something "matching" or "not matching". It's just a cosine similarity value. To determine if an image "matches", you have to check if that similarity is within some given threshold. If the results of the cosine similarity operation are encrypted (they are with HE), that wouldn't be possible to determine.

The bigger privacy risk would be that the device routes the request to a specific database shard based on whichever has a center-point closest to the image embedding on the device. They take steps to protect this information such as third-party proxying to hide user IP addresses, as well as having devices send fake requests so that the server cannot tell which are real user data and which are fake data.

Reply on news.ycombinator.com