Presumably at this stage is where malicious hashes would be detected and removed from the database.
An unaccountable "Apple Employee" who is likely (in the US and other countries) to be a LEO themselves will see a "visual derivative" aka a 50x50px greyscale copy of your content.
There is no mechanism to prevent said "employee" from hitting report 100% of the time, and no recourse if they falsely accuse you. The system is RIPE for abuse.
>Presumably at this stage is where malicious hashes would be detected and removed from the database.
Collision attacks have already been demonstrated. I could produce a large amount of false positives by modifying legal adult porn to collide with neural hashes. Anyone could spread these images on adult sites. Apple "employees" that "review" the "image derivatives" will then, even when acting honestly, forward you to prosecution.
Of course there is. The judicial system.
(Although, to be clear. I don't live in America and I might be more worried about this if I did.)
The recourse should be before this reaches the law.
1) Spent who knows how long in jail
2) Lost your job
3) Defaulted on your mortgage
4) Been divorced
5) Had you reputation ruined
Money can't fix everything, and trusting the courts to make you whole years after the fact is a foolish strategy.
How, if 1) the original content is never provided to Apple, and 2) the offending content on consumer devices is never uploaded to Apple?
This entire system was a way for Apple to avoid decrypting your photos on their servers and scanning them there.
Hypothetically, if Apple implemented this system and switched to E2E for the photo storage, you'd be more private overall because Apple would be incapable of seeing anything about your photos until you tripped these hash matches, as opposed to the status quo where they can look at any of your photos whenever they feel like it. (And the hash matches only include a "visual derivative" which we assume means a low res thumbnail.) I say hypothetically because Apple never said this was their plan.
You can argue about whether or not Apple should be doing this. But it does seem to be fairly standard in the cloud file storage industry.
I suspect the key would be that there'd be a team verifying that something is actually child pornography, because the system is a perceptual hash rather than a strict comparing-bytes so before someone looks at it they're not certain.