This, combined with human error during the manual review process might result in someone getting reported. Seeing as twitter (and other social media sites) jump on the bandwagon whenever someone gets accused of being a pedophile, this might destroy someones life.
The entire story might seem a bit to far fetched, but based on past events, you never know how bad something 'simple' as a hash collision can be.
You can end the conversation right there. If you are up against a state actor, you have already lost.
It's always possible before, but client-side CSAM detection and alerting has weaponised this.
Previously, you always had to somehow alert an unfriendly jurisdiction. Now, you just use malware like Pegasus to drop CSAM, whether real or disturbed from legal porn, and watch as Apple tips off the Feds on your enemies.
But now China can send some legal pornography (eg closeup pussy pictures), disturbed to match a CSAM hit, to a journalist they don't like and get them in jail.
Why can't China do this before? Because previously, they'd still need to tip off authorities, which has an attribution trail and credibility barrier. Now, they can just use Pegasus to plant these images and then watch as Apple turns them into the Feds. Zero links to the attacker.
I know of zero instances of this attack being executed on anyone, so apparently even though it's been possible for years, it isn't a material threat to any Apple customers today. If you have information to the contrary, please present it.
What new attacks are possible upon device owners when the CSAM scanning of iCloud uploads is shifted to the device, that were not already a viable attack at any time in the past decade?
The low res derivative will match, perhaps even closely, because pussy closeups look similar to an apple employee when its grayscale 64 by 64 pixels (remember: it's illegal for Apple to transmit CSAM, so it must be so visually degraded to the point where it's arguably not visual).
The victim will get raided, be considered a paedophile by their workplace, media, and family, and perhaps even go into jail.
The attacker in this case can be users of Pegasus unhappy with a journalist.
1. accepted by innocent user,
2. flagged as known CSAM by NeuralHash,
2b. also flagged by the second algorithm Apple will run over flagged images server side as known CSAM,
3. apparently CSAM in the "visual derivative".
That strikes me as a rather remote scenario, but worth investigating. Having said that, if it's a 3-letter adversary using Pegasus unhappy with a journalist, couldn't they just put actual CSAM onto the journalist's phone? And couldn't they have done that for many years?
There's a pretty good chance that it was inevitably going to get expanded to handle pictures arriving at the phone through other means.
For one, you can't know if that's true as the image could have been manipulated to appear as such. For example you wouldn't know if a kind of steganography has been used to hide image in an image and that neuralhash picked on a hidden image.
> but I don't understand how they can get it on someone's phone
There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc...
How would NeuralHash pick a "hidden image"? It only uses the pixels of the image to get the hash. Any hidden image in the metadata would not even be picked up and no amount of steganography can fool NeuralHash.
> There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc...
As iterated elsewhere in this thread, random gibberish pixels colliding with CSAM would definitely not be useful in incriminating anyone. The manual process would catch that. Also, if the manual process is overloaded, I'm pretty sure basic object recognition can filter out most of the colliding gibberish .
The NeuralHash would "see" the planted image, but for the viewer it would appear innocent.
I am trying to say that a person reviewing image manually, without special tools will not be able to tell if the image is a false positive and would have to report everything.