I read in other posts there's some sort of review process and a way to verify if the image is a match or a collision (I don't know much about he details) but I read the latest posts about the attacks that were being worked on.
I mean, with the complexity these attacks have I think it'd be easier for a ransomware gang to just infect you, plant the CSAM, find reliable contact info, verify it, lock your phone and extort your through an untrackable side channel (if this system didn't exist) or something a bit more elaborate / targeted (not even at NSO level).
IDK, I think the phone burns battery for dumber reasons at a higher rate, this should only be activated when there's a new picture written to disk and it's probably less expensive resource wise than whatsapp / telegram / imessage checking for new messages periodically don't you think?
I think if they don't royally fuck the process up or turn it into some idiotic fake way of getting the cops whatever paperwork they need to force you to give them access to your files it's a good thing.
In your example about planting CSAM, why would on/off device matter since the new feature only checks for items going to iCloud anyway? The planting CSAM attack vector is available right now for any device connected to FB, OneDrive, or Gmail, and I don't think planting material has been an issue.
If you're hit by an NSO client and they have an agent running in your phone checking in with their C2, what do you think would be easier :
1 - Run a reverse proxy in your phone, steal your credentials (or session data) and use that connection to upload the material 2 - Write it to disk and wait for the media scanner service to pick it up and act on it?
I mean, in the end it's not about the technology but the people operating it, if apple is really incompetent and law enforcement is shitty as usual then yeah, people might end up behind bars for no reason, which sucks but in that case I think the focus shouldn't be the technology itself but how shitty and unfair the system is.
Using Apple devices used to be all about how they serve the user to bring joy. Knowing they now spend even a single cpu instruction on trying to frame the user turns the device from something I loved to something I fear.
All the talk about human review and multiple failsafes does not smooth things over. App store review is a prime example of how their review process can be seriously flawed - scam apps and subscriptions sometimes even being FEATURED in editorials on the app store.
It does not matter that you will be found innocent in the end. Just being put under investigation for csam can make anyone's a life living hell. Getting your AppleID blocked, even if temporary, can cause severe problems.
When a company advertises that "what happens on your phone stays on your phone", and then proceeds to build snitchware into the phone that reports on received imessages to the "family head of household" and reports and UPLOADS photo roll items that were never intended to be shared, to human review, well... that company does not appear to be honest anymore.
In addition, this only works for <18 accounts. If the abusive figure goes as far as making other family members recreate Apple IDs and lie about their age every 5 years to keep getting access to iMessage (and other parental controls like screen time) then there's not much Apple can do.
I have read their papers, I understand the system and the safeguards they put in place, but none of them are good enough to have scanning on my device. There is nothing that is good enough. On device scanning for "illicit" content is a box that cannot be closed.
IIRC it's deeply entrenched in the system and no one reversed their way deep enough to be able to replicate it. Now this might sound silly, but it's just an example, a contrast maybe of how the hard work the people behind asahi are putting or the huge jailbreak community, but the idea I'm trying to convey is that the playfield is HUGE and they just don't need this.
The one thing I would be 100% concerned about is the investigation process for matches because that's mainly where human interaction and decision making com into play and we humans SUCK, we've put people behind bars for years for no reason and with all this AI crap there have been a lot of news articles about that kind of stuff and that's something we should definitely be worried about, but I guess it's less about the tech and more about the people in charge right?
I also don't agree that the human-in-the-loop part of the process is a/the problem. Are you suggesting that it should just send the findings straight to the FBI... where a human would review it? Or maybe skip all of the messy middle part and if the model detects enough CSAM just send an APB to the local police to pick you up and take you straight to prison with no trial?
So, the idea is that even though it's not an impenetrable fortress, there are still plenty of dark places to introduce subreptious changes.
As for the human-in-the-loop part, I don't know why did you get so snarky, what I was talking about was that this is the layer that should be scrutinized the most, all of those components because even without the technology those are the people that will put someone in jail with no verifiable evidence.
That is a huge difference.
I got snarky because the human-in-the-loop for decision making is the the least concerning part of the process and the alternatives are as ridiculous as I laid out. There will always be a human-in-the-loop in this process - I'd rather it start with Apple's human, then law enforcement, then a prosecutor, then a judge, etc.