Comment by lamontcg | Hacker News Reader

Comment by lamontcg | Hacker News Reader

> But when a backdoor is installed, the backdoor exists and history teaches that it’s only a matter of time before it’s also used by the bad guys and authoritarian regimes.

Problem is that this scanning is necessarily fuzzy and there is going to be a false positive rate to it. And the way that you'll find out that you've tripped a false positive is that the SWAT team will knock your door down and kill your dog (at a minimum). Then you'll be stuck in a Kafkaesque nightmare trying to prove your innocence where you've been accused by a quasi Governmental agency that hides its methods so the "bad guys" can't work around them.

It isn't just "authoritarian regimes" abusing it, it is the stochastic domestic terrorism that our own government currently carries out against its own citizens every time there's a beaurocratic fuckup in how it manages its monopoly on violence.

This is the "Apple/Google cancelled my account and I don't know why" problem combined with SWATing.

Stochastic domestic terrorist is not a term I ever expected to read, but it seems very fitting for the “data-driven” (but generally poorly evaluated) future we find ourselves drifting into.

"Stochastic terrorism" has been a popular term for a while[0], but it usually refers to the actions carried out by individuals, not government agents. Ironically, though, the word "terrorism" itself originally referred to the actions of a tyrannical government against its own citizens[1], so perhaps the definition has come full circle.

[0] https://www.dictionary.com/e/what-is-stochastic-terrorism/

[1] https://www.merriam-webster.com/words-at-play/history-of-the...

This really has been decades in the making, hasn’t it? Apple is only adding the latest piece of the puzzle. The seeds of everything you’ve described were planted years ago.

How did we get here? How did everything become so politicized and polarized? How did law enforcement become so militarized? How did we as a society become so terrified and distrustful of our neighbours?

>Problem is that this scanning is necessarily fuzzy and there is going to be a false positive rate to it. And the way that you'll find out that you've tripped a false positive is that the SWAT team will knock your door down and kill your dog (at a minimum).

Not true. Hash matches are to be human reviewed. So no, people won't get "swatted" accidentally as you allege.

The other concerns people have been voicing are certainly valid though (IMHO).

>Hash matches are to be human reviewed.

Until it proves too expensive, then a different AI system will do it instead. I have zero faith that it'll be a fully competent, well trained, well rested, well paid person will actually be doing these reviews in the long run.

If you actually think there is going to be a fully automated system dispatching police SWAT teams throughout the US without a manual (or judicial) review then.... I really don't know what to tell you.

It'll all be shadow-ban type stuff, your account will be turned off, without appeal, things of that nature.

That's a different thread of comments. This one is about automated dispatching of SWAT teams.

Guess its hard to believe that their is a world outside of the United States. Already do whatever it takes to pin some BS on someone that opposes them and has them executed. This would be no different.

I don’t understand. If the government can just claim you had CP without proof, then they can just… do that? They don’t need Apples system.

Especially since anything flagged by this system is manually reviewed by Apple. So, there would exist counter-evidence for the govt claim.

Don’t misunderstand, I see how this system is ripe for abuse. I was just commenting on the specific claim that there will be automated SWAT call outs (presumably in the US).

It should be extremely easy for any group that has access to tools like NSO Group's to use the information on a device to convincingly add data to devices. What is generally missing is the question of why certain people's devices were selectively searched. Attacks by the west on journalists often rely on the excuse of border searches for example.

> If the government can just claim you had CP without proof, then they can just… do that?

The problem is that they can do whatever if you have CP. Emphasis on "have": you might not even know you have it, because all it takes for you to be guilty is for a forbidden bit of data to be on your disk or cloud account. How did it end there? Doesn't matter much. It's unlikely you'll be convicted if someone else maliciously puts CP on your drive, but it's extremely likely your whole neighborhood will know about it before it's settled. Think about that.

> They don’t need Apples system.

Apple's system makes it sure that whenever someone puts CP on someone else's account, it rings the police and starts the nightmare. Without it, there's a few extra steps that make the nightmare much less likely to happen in the first place.

This is plausible and exactly why I oppose this system.

The only argument I made was against the claim that fully automated police dispatches will occur because of this system (in the west).

Making outrageous, tinfoil hat level claims does not help our collective argument against this system.

It just makes us look like paranoid conspiracy theorists.

Let’s make coherent arguments (like the one you made above) instead of fantasy ones.

Yeah, it normally requires an anonymous phone call from a VoIP number.

Is this an allusion to an actual occurrence?

https://en.wikipedia.org/wiki/Swatting

A dodgy phone call from an unknown number abroad by a stupid teenager is enough to get a SWAT team to murder innocent civilians, what makes you think they'll be smarter when the "call" comes from a robot?

I mean, seriously, you're assuming as beyond obvious something that's demonstrably wrong RIGHT NOW.

Those prank SWAT calls are (a) from a human and (b) based on the assumption of IMMEDIATE risk of death/injury. Furthermore, (c) the CP scanning implementation itself requires human review.

So no, under the proposed implementation (and current judicial requirements in the west) robots dispatching armed police to kill pets or people without ANY human oversight because of a situation that is not time sensitive is mere fantasy.

Don’t misunderstand, I am against this whole thing for the reasons many others have well articulated in this thread.

But the claim that robots are dispatching armed police without any human interference RIGHT NOW is provably wrong.

Show me one incident where this has happened already and I will donate $100 to the charity of your choice.

And yes, the judicial rules and Apples implementation could change in the future. That’s certainly a risk but is not the case RIGHT NOW. I mean, seriously?

I have to imagine that you’ll be investigated and can let Apple rehash your photos and find the ones that tripped the system. If Apple rehashes all your photos and suddenly none of them are tripping, and they accuse you of deleting the bad ones (assuming you haven’t deleted anything) the problem isn’t the hash algo, it’s something else in the system.

That said, I don’t like my phone being a snitch.

Or Mechanical Turked.

Eh, humans make mistakes, that may reduce the false positives but if this program runs long enough, I'm sure it will happen to someone when someone accidentally presses the wrong button.

Google and Apple have burned all their trust for this kind of thing. They already do terrible with all the people who get banned off their platforms with no notice of what they did wrong and no recourse. These companies cannot be trusted with the power to arrest you. They don't care if they ruin innocent lives due to edge conditions.

> Hash matches are to be human reviewed.

This comment suggests this phrase might be cleverly worded, to make it seem like the images are human reviewed, while that actually not being the case: https://news.ycombinator.com/item?id=28096059

Reviewing potential matches sounds like an awful job. Retention might be even lower than in those FB abuse centers.

That's part of why possession is illegal. The material is toxic.

This is wrong - the iCloud check is against known CSAM hashes, the false positive rate is essentially zero.

One in one trillion chance per year, per the paper on the Apple site.

Yeah, that’s essentially zero.

I see the 'one in one trillion' but it's a bit vague; I read it as '1 in trillion chance' per image.

So when we have a billion iPhones in the wild taking 10 images a day...1 in a trillion chances happen every few months. Now, if that triggers some further review, maybe that's an acceptable false positive. If it triggers a SWAT team, I don't think it is.

They have also indicated that a single match won't be enough to trigger it, so an accidental match (being that 1 in a trillion person) is not enough.

I would like to bet Apple's market value against that number being correct in an adversarial context.

This seems naive, there are always false positives

It’s not naive, it’s math.

It's not a math problem, it's a human problem.

suppose you have a partner who is a 'petite' woman of 34. She enjoys posting nudies on a website, but without her face in that picture. Someone who collects child porn downloads it, because he enjoys that picture. A year later he gets caught by the police and all his pictures get marked as 'verified child porn'. Suddenly you get marked as owning child porn.

That isn’t CSAM.

https://www.nytimes.com/interactive/2019/09/28/us/child-sex-...

Apple’s thing has some sort of threshold anyway so one image would not trigger it. I don’t buy your example - the CSAM images are not what you’re describing.

Who determines whether something is CSAM? How do you or I know that every single one of those images actually is CSAM? How do we know if the FBI or CIA or CCP adds hashes of innocent pics in order to pin a crime against someone?

No. You can design a system where the FPR is essentially zero. Even if shitty md5 is used.

Perceptual hashes are not exact hashes, otherwise they would be useless for this task; you would just mirror the image or change 1 pixel.

They are instead fuzzy classifiers, and thus have non-zero error rates.

But this trigger requires many of these false flags to exceed the threshold (most likely why it exists). I imagine the "one in a trillion" numbers they claim for false flagging rate are probably cemented in reality, and make it trivial for human review.

I've explained the problem here: https://news.ycombinator.com/item?id=28099927

How does this relate to the allegation I was responding to? There is a difference between false positives and false negatives you know. False negative is where the criminal evades the system through measures you describe. False positive — what I was responding to — is where Apple incorrectly accuses an innocent person and thus violates their privacy during the subsequent review.

> No. You can design a system where the FPR is essentially zero. Even if shitty md5 is used.

How can you do that, considering md5 can have collisions?

A confusing thing that I think people haven't addressed clearly for the most part:

For a hash (whether cryptographic or perceptual), there is a chance of random collisions and also a difficulty factor for adversarially-created intentional collisions. The random collision probability has to be estimated based on some model of the input and output space (with cryptographic hash functions, you would usually model them as pseudorandom functions and assume that the collision probability is the same one created by the birthday paradox calculation).

Intentional collisions depend on insight about the structure of the hash function, and there are also different kinds of difficulty levels depending on the nature of the attack (preimage resistance, second-preimage resistance, and collision resistance). Gaining more insight about the structure of the hash function can act to reduce the work factor required for mounting these attacks. That should be true for perceptual hashes just as much as cryptographic hashes, but presumably all of the intentional attacks should start off easier because the perceptual hashes' threat models are weaker and there's much less mathematical research on how to achieve them.

And in AI systems involving classifiers, it was generally easy for people to create adversarial examples given access to the model. Perceptual hashes for estimating similarity to specific known images aren't the exact same thing because it's less like "how much like a cat is this image?" and more like "how much like NCMEC corpus image 77 is this image?", but maybe some of the same techniques would still work. In the cryptographic hash analogy, I guess that would be like trying to break preimage resistance.

Thanks for the detailed explanation. I understand why that works for perceptual hashes if you make them really precise, however I doubt it would work with md5, which is why I asked.

The discussion I thought we were having was about false positives and not adversarially induced false positives. For the former the random collisions have a probability of 1/(2^64).

To mitigate adversarial false positives one idea is to use the combination of a cryptographically strong hash along with a randomly selected perturbation of the file. Prior to hashing, perturb the file and submit both the hash and the selected perturbation to apple. Apple selects the DB based on the perturbation and proceeds with matching and thresholding.

I thought that the random collision for md5 was way higher than that. If it's that low, you're right, this would work. I'm not sure I understand the part about the pertubation.

It’s actually 1/(2^128).

If the attacker does not know how the image will be perturbed prior to hashing then he cannot generate an image which matches with known CSAM.

> The discussion I thought we were having was about false positives and not adversarially induced false positives.

I think the rest of us have been discussing how this can and will be abused, by definition by adversaries.

Many of us have also observed for years how systems are abused so we sadly have a gut feeling for this.

> the false positive rate is essentially zero

I highly doubt that

Based on what? Unless you point out a flaw in the math, it’s zero.

> Based on what?

Based on the fact that ultimately we can't check the system. And based on the fact that at some point in the chain humans are involved. [1]

What we are left with is "trust in Apple" not "trust in math".

[1]: https://news.ycombinator.com/item?id=27878333

Yes but my understanding is that the hashes are perceptual, as opposed to cryptographic.

If the system was matching against known cryptographic hashes the collision / false positive rate would be small, but the fuzzy matching involved with perceptual hashing necessarily has a greater false positive rate.

And that doesn’t even begin to address the detection of sent and received “explicit images” which are detected on device and don’t have a set of known hashes.

I suspect that's why they have some threshold that moves the false positive rate to one in one trillion.

The iMessage bit is different - it's only on device, only on child accounts, and only alerts parents. It's more akin to a parental control feature than anything else.

This is a great point, thanks for adding that detail.

It was pretty dumb of Apple to announce both of these things at the same time.

They have nothing to do with each other, and I've seen a dozen people on HN confuse them. If the message is getting muddled here, it will be hopelessly conflated in less technical circles.

I'm concerned and upset about the CSAM filter for all the reasons that keep hitting the front page, but don't care about the opt-in parental controls at all, and if I had kids, I might want them.

But if I thought the CSAM filter worked like the nudie-detector filter, I'd be wigging out.

It doesn't matter if the cryptographic algorithm is one in a trillion. If there's a concurrency bug in the application which applies the algorithm the wrong account could be flagged against a legitimate image in the database. If the human involved overly trusts the system they may not validate against the actual file in the users account, or may assume the user deleted it somehow and figure its "safter" to let law enforcement figure it out.

Bugs in the application of the code, combined with human complacency and mistakes can certainly lead to errors, even if the cryptographic algorithm itself was perfect.

We really need to bring back comp.risks

> I suspect that's why they have some threshold that moves the false positive rate to one in one trillion.

So now instead of sending just one nice innocent very high resolution images of "Tokyo City" or something with something horrific hidden somewhere you have to send a few such images.

That is reassuring. I can never believe anyone except me will think about that.

(If the system is too dumb to detect this it is worthless, and if it is smart enough this opens the floodgates for anyone wanting to make trouble for just about anyone.)

This would require multiple hash collisions.

No need for hash collisions, that's what makes it so bad: just edit multiple known verybad images in multiple nice high res photos like I tried to explain above.

It’s a visual hash, not something like an md5. And now you’re talking of a directed attack to intentionally make hash collisions?

This isn’t a normal file hash. It’s not SHA or bcrypt. A wide variety of states (1s and 0s) can have the same hash.

The idea is to take an image and have all of its possible derivatives create the same hash.

For example, if a hash was made of the Mona Lisa, any copy no matter how large, small, black and white, would have the same hash.

Think of all the ways the Mona Lisa could be transformed and still be the Mona Lisa.

The combinations of ones and zeros would be in the billions. If not more.

And all those possible combinations of ones and zeros go back to the same “hash.”

That’s extremely resourceful intensive.

My guess is that they are going to transform the images into a very low resolution, black and white thumbnail. Then compare it against known abuse images that have been similarly transformed.

Or they’re using AI. They might be using AI.

Either way, it’s guesswork. How many images might be transformable to the same black and white thumbnail. I don’t know.

Reply on news.ycombinator.com