Hacker News Reader

Comment by cwoolfe | Hacker News Reader

It is possible to detect CSAM in end-to-end encrypted messaging by doing it on-device on the client side before it is encrypted and/or on the receiving client side after it is decrypted. iMessage already does this in the latest release. Most smart phones have AI-enabled chips that would be able to run images/videos against classification algorithms. The tricky part would be enforcing that users use clients which do this, so the task becomes regulating allowed messaging clients, which might be impossible. That being said, one could probably knock out 80% of the problem by legally forcing the hand of all the major platforms to do client-side scanning. At that point, only the truly dedicated would move their comms to another platform. Mobile users are very susceptible to nudges.

View on HN

I think the problem is different. This is not about a technical possibility. This is about who will submit and manage the CSAM fingerprints. With the technology it will be trivial for oppressive governments to query which citizens have a copy of a particular image on their phone.

Consider a scenario:

- you use whatsapp, end-to-end encrypted by an USA company

- someone, perhaps a random stranger, sends you an image in which there is a criticism to a ruling party in Dabujistan

- your whatsapp has turned on the feature to save images on your phone

- you happen to transfer via Dabujistan

With CSAM tech, and fingerprints managed by the governments, you might be subject to jail.

> This is about who will submit and manage the CSAM fingerprints.

Not it's not. It's about "Why the fuck should I be under constant surveillance?"

Our computers are practically brain extensions at this point. I don't want anyone wiretapping my thoughts or coming close to it by wiretapping my private conversations with my family. Fuck off.

The unfortunate part about a centralized CSAM list is that now you need a committee to review every CSAM addition (not just the hashes but the source material to confirm it's CSAM and not political) and the committee has to have enough people to make sure the committee isn't all in kahoots

iMessage does not do this in any release, that plan was abandoned due to legitimate privacy concerns. The feature that is implemented is a parental control that can be enabled on the devices of minors, that blurs any detected nude images with a warning 'do you want to see this'.

It's not possible to detect CSAM without breaking the security model of the system for everyone, if the system can make any kind of report of detected content to an outside entity. If the system scans messages client side and then sends anything to do with the message to authorities, that is breaking the end to end encryption.

This is either done by 'AI', which means huge numbers of false positives that are actually legal, private and probably intimate get sent to authorities (hence breaking the end-to-end security model for that legal content), or by a secret, un-auditable collection of hashes - which are still fuzzy matched, so there would definitely be false positives (again, breaking the security model for legal content), but potentially also Governments could start to add other, legal material to the database, such as certain political material to detect dissidents. Since it's un-auditable, there is no way to know whether it's actually just illegal material in the database.

So whether it's AI or PhotoDNA or whatever, there's no way to do it that doesn't break the security model and cause more danger to innocent people, including destroying the privacy of children as well as adult, law-abiding citizens.

It is worth noting that due to endpoint key escrow on by default in the non-e2ee iCloud Backup, iMessage is already presently broken and not e2ee as Apple has endpoint keys available to them to decrypt almost all iMessages that transit the service (Messages in iCloud) in realtime.

Even if you enable iCloud backup E2EE, or disable iCloud entirely, iMessage is still not E2EE as your conversation partners are escrowing (to Apple) the keys from the other endpoint.

Why should my phone be acting like I'm a criminal? That's like searching the bags of everyone who exits the supermarket, or police doing door-to-door house searches "just in case". What's next? Verify every photo I take with my phone?

Or make everyone go through metal detectors and millimeter wave scanners at the airport, take their shoes off, treat any liquid over whatever the small limit is like it’s bomb-making material (but still let that person fly; checkmate, would-be bomber!)…it’s all security theater. Notice how they don’t release their testing rates on how many of the test weapons and threats make it through TSA security, because then it would be too easy to point to that and rightfully assert that they utterly fail at their mission while wasting untold lifetimes worth of human time and make travel that much more miserable. https://www.schneier.com/blog/archives/2015/06/reassessing_a... (2015) is a good read on that if you don’t already know.

My point is that when politicians do something “to save the children”, it’s rarely about actually helping anyone but themselves.

Yeah that's what this person wants, to scan every photo you take...ick.

If the app is processing data in a way you do not want before sending it, especially one that will no doubt be constantly argued to scan for more and more things they find objectionable, then it defeats whole point of end to end encryption, and you might as well just not do it at all.

What classification algorithm can detect CSAM, given that humans cannot reliably determine someone's age? [1] [2]

[1] https://www.eldia.com/nota/2010-4-23-actriz-porno-salva-a-fa... [ES]

[2] anyone who still gets ID'd for alcohol despite being well over 18, and often over 25 too

Yeah, no one who’s actually had to work with this on a technical level would seriously suggest our current GPT-4 level of AI for detection. It’s hash-based with some sugar on top.

Do you imagine that a significant proportion of CSAM is shared by users who are so casual about it that they would be "very susceptible to nudges"?

I’m not sure if it’s a happy or sad thing how unaware most people, even the technically-savvy crowd, are about the extent of human trafficking and slavery in 2023. Is it proof that civilization is working how so many don’t need to concern themselves with hunting and gathering, fighting wars, fighting crime, or fighting in general to survive, and are free to spend their time on recreation and the pursuit of happiness instead of survival?

I suspect that a large portion of CSAM is nude selfies that might be susceptible to "nudges" in the behavioral economics sense.

This may well be true according to the legal definition, though a system that matches against a fixed list of hashes, perceptual or otherwise, rather than working on magical AI, would not catch those. However, if this is all that the measure would be effective against then this is rather at odds with the proponents' narrative which makes this out to be about pictures produced and shared by adults with criminal intent.

To hell with your "nudges". It It's a cutesy way of saying "flex of State power", and everyone effing knows it.

No. It's not appropriate.

I think that it's only going to get faster, easier, and more accurate to generate sexualized pictures and videos of someone younger than 18.

Whether you or I consider that "real" CSAM, is moot. What matters is that more and more girls are going to be deepfaked into content they didn't consent to (and in fact can't consent to due to their age.

The proliferation of that content will lead to new laws driven by parents and politicians eager to "think of the children". However impractical, I could see image generators being treated like drugs or firearms and being highly restricted or banned.

Please don’t AI detect CSAM. That is a disaster waiting to happen.

It works great! Until it doesn't, and then someone gets put into jail and the court doesn't bother reviewing the AI's verdict because how could it go wrong?

It's not that you would end up in jail (although granted, that level of fuck-uppery is possible), it's just that anyone flagged would go through hell with real-world consequences before everything is cleared up.

Even if no one charges you it still creates a mess for the accused. We already saw how google handled a verified false positive scan that the police determined was innocuous.

And do not hash detect it either. If you mass scan eventually you are gonna hit a collision .

A greater point here is that end to end encryption in this context is almost entirely a lie.

How do you plan to do that without having a database of known CSAM on the device?

Reply on news.ycombinator.com