The fact of the matter is that unless you possess a photo that exists in the NCMEC database, your photos simply will not be flagged to Apple. Photos of your own kids won't trigger it, nude photos of adults won't trigger it; only photos of already known CSAM content will trigger (and that too, Apple requires a specific threshold of matches before a report is triggered).
[1] "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account." Page 4 of https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
Probability of N false positives (assuming independence) = p^N
Threshold N is chosen by Apple such that p^N < 10^-12, or N log p < -12 log 10, or N > -12 log(10)/log(p) [since log(p) < 0, since p < 1].
ETA: Suppose, just for the sake of the argument, that p = 10^-3 (one false positive in 1000, so really quite bad).
Then log(p) = -3 log(10), so N > -12 log(10)/(-3 log(10)) = 12/3 = 4.
Similarly, if p is one in a million (10^6), then N would be required to be > 12/6 = 2.
In practice, I'd expect N to be larger than 4, in other words, Apple being very conservative here.
ETA: The above doesn't take into account how many images M you have. The analysis gets more complicated, but N needs to be way larger than 4. I'll think about it some more.
The parent poster does make the mistake of assuming that other pictures of kids will likely cause false positives. Anything could trigger a false positive - especially flesh tones. Like, say, the naughty pictures you've been taking of your (consenting) adult partner. I'm sure Apple's outsourced low-wage-country verification team will enjoy those.
I'm not sure they'll be able to after looking at CSAM all day...
This would be a great job for an actual pedophile.
There was a good article [0] that was on HN a couple days ago that touches on the flat out lie regarding "one in a trillion" and how PhotoDNA sounds poorly thought out.
[0] https://www.hackerfactor.com/blog/index.php?/archives/929-On...
The details for those processes hasn't been fully disclosed, and it isn't possible to say whether 1 in a trillion is a reasonable estimate or otherwise.
Don't you think Apple has independently arrived at that very same possibility? Maybe even thought about it and extensively tested it? Then put a probability on it? And then chosen the threshold such that the overall probability of falsely flagging an account is, indeed, minuscule?
Beyond that, assume that a false positive occurs and the innocent person is taken to court. Why would they have to fear being convicted if they don't actually hold any incriminating evidence? At most, that would become evidence against using perceptual hashing in future court cases.
The issue in that case is the violation of the innocent person's privacy, not that they have a risk of being falsely convicted. The courts would still need admissible evidence, and I don't believe that only having a perceptual hash and a set of legally photographed images clears that bar.
However, it becomes a completely separate issue if the false positives are "coincidentally" used to persecute marginalized groups in other countries where the same set of laws don't apply. But Apple has stated that they have no intention of expanding the system's scope to follow those laws. There isn't any evidence yet that Apple will do such a thing, or that they've already done it in the past. We are free to disbelieve them, but that's what they've stated. We can only hope that they won't change their minds.
Because going to court is extremely expensive?
Because retaining legal council is extremely expensive?
Because your district attorney will likely inflate the charges to coerce you into taking a plea deal, despite your innocence?
Because you don't want all of your private documents, including ones completely unrelated to the alleged crime, entered into the court records?
Because a "jury of your peers" can be convinced to believe just about anything?
Because even if you're acquitted, most people will still believe you're guilty, and treat you as such?
There are a myriad of reasons one shouldn't let the police go on mass criminal fishing expeditions. A lot of innocent people take collateral damage as a direct result of the incentives involved in an adversarial justice system.
Because a case like this, you'll be paying a lot of bail just for the privilege of defending yourself properly. (Might have to sell/mortgage your house, or your mother's house).
Because you will probably make the news, and the false accusation will be on the internet forever.
Because you will probably get the living shit beat out of you by the arresting police.
Because you will probably get the living shit beat out of you by your fellow inmates.
Because the incompetent people who charged you can't be sued due to qualified immunity, so no skin off their back.
Because your family will have a lot of stress on it and this will affect your spouse, children, siblings and parents in a very negative way, probably for the rest of their lives.
Because you've lost your job in the process.
Because you'll probably never get a decent job again.
Because you'll probably need a lot of psychological counseling assuming you survive this process.
Positives get reviewed by humans, at which point false positives are identified and discarded. We would not hear about them, and there would not be any reporting about them in the press.
You might think that this is the system working as intended, but I do not and would never consent to Apple giving my photos to some random anon to look at.
Let's not forget, this part only happens if you are lucky. You get the wrong photo reviewer and you catch your girlfriend with the wrong type of lighting that makes her look as if she could pass for 17 and you're going to have the police show up at your work and cuff you, demanding to know who this person of interest is. Adult males have been tried for possession of CSAM for having images of well-known absolutely above 18 adult video stars: https://reason.com/2010/05/03/porn-star-saves-man-from-incom...
Article here: https://apple.news/AhVzAY--DT36Oz22W7taL9Q
For now.
"If you're concerned that a hurricane might blow your house over some day, you shouldn't build one today."
"If you're concerned that your child might grow up to be a psychopath, you shouldn't have kids."
How is Apple going compensate the innocent people?
>The issue in that case is the violation of the innocent person's privacy, not that they have a risk of being falsely convicted. The courts would still need admissible evidence, and I don't believe that only having a perceptual hash and a set of legally photographed images clears that bar.
So... You admit that this type of scanning is an invasion of privacy, and would likely be a flagrant constitutional violation if done by the Government?
So why is it okie-dokie for the private sector to do this type of systematic check and balance evasion? That's what gets to me.
Then where are their published peer reviewed papers demonstrating their false positive rate? The burden of proof is on them and they don't have any verifiable public data at all.
"Using another technology called threshold secret sharing, the system ensures that the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images."
"The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. This is further mitigated by a manual review process wherein Apple reviews each report to confirm there is a match..."
And when the manual review process sees that the images flagged aren't NCMEC classification A1 (A=prepubescent, 1=sex acts) the flag is cleared.
What's relevant is the overall false positive rate. If they require 6 matches for example, it's enough for each match to have a 1% false positive rate in order to get 1 in trillion overall.
So if Apple has a "1 in a trillion" 1e^-12 chance of flagging a person and they require (my guess) 3 hits to flag someone, then that would mean the chance of each single image being false positive is (1e^-12)^(1/3) = 1e^-4. So that means:
Expect 1 in 1000 images to be uploaded to Apple for potential human verification.
But the truly sad thing is that there are trillions of instances, photos, moments that will never even see the light of day. and plenty of human children get abused, raped, murdered every single day.
Child abuse should be a capital crime.
This is a pandaora's box of trust. Once you open it, you have to trust in perpetuity.
I hope you won't throw the "slippery slope is a fallacy" fallacy at me.
Does this change to the software code represent a slippery slope of motive or opportunity? Many here have said yes, but in my opinion, no. When software can update itself, every single update is an opportunity for the software to betray you. That risk is already high; the risk profile doesn't increase because a particular change feels slippery slopey to you.
As soon as any closed-source software implements automatic software updates, you've always one malicious update away from the system betraying you. Interim steps are unnecessary. Whether it's Chrome, or Firefox, or Windows, or Android. Heck, even Ubuntu. Any of them could betray you at any time. Potentially trash their reputation in the process, but that's a mere technicality.
Therefore the slippery slope is the wrong metaphor. The correct metaphor is trust. Does this change lower my trust in Apple? Me personally, no. If anything, Apple's transparency has increased my trust in them. It gives me confidence that Apple won't use the fear of bad PR as an excuse to conceal serious things like this.
It is currently considered completely unacceptable for companies to scan the data on the user's own disk.
If someone wanted to start doing that, they would have to create a plausible and convincing excuse.
Once people accept that excuse and time passes, they can slowly "expand" the territory of this excuse (push the overton window further).
"Protecting the children" is exactly this kind of plausible and convincing excuse.
It needs not be the case that Apple specifically wants to scan user data so they came up with this excuse.
It's simply that, once scanning users data "for the good of humanity" becomes acceptable, _some_ malicious actor will push the overton window further and further.
Imagine in 10 years from now, all operating systems will scan users data to detect potential child porn material. It might even become required by law, or just by "social pressure". Just like it is now almost required by "social pressure" that social media platforms censor discourse and information.
It's then very easy to expand this capability to also detect illegally downloaded music, or adult videos, or whatever deemed unacceptable.
Highly technical, libertarian-minded people see a big distinction between on-device and off-device scanning. But such people have always eschewed the "managed experience" model of Apple devices.
That is the goal of all perceptual hashing algorithms.
> Apple's system only concerns exact hash matches
Then it is almost useless. All someone would need to do to evade the system is make minor enough adjustments to illegal images so that the distance between resulting hash and the original is minor.
In reality, perceptual hashing systems use fuzzy matching on hashes by using a metric like hamming distance to calculate differences between two or more hashes.
I'm sure you're in a prime position to judge that.
> In reality, perceptual hashing systems use fuzzy matching on hashes by using a metric like hamming distance to calculate differences between two or more hashes.
Again, Apple's NeuralHash purposefully doesn't work like that [0].
[0] https://twitter.com/jonathanmayer/status/1423370142411476993
I can believe that some people might be stupid enough to believe that a private Facebook group was secure. But who the hell co-mingles their deepest darkest dirtiest secrets with pictures of their family and last night’s dinner?
What for parents is "children playing in the pool in the garden - shared on YouTube so grandparents could see it"
- is something that is collected into playlists and shared with "interesting" timestamps in certain circles. A story here on HN a couple of years ago about a (for us normal people) totally innocent video having reached a million views or something on YouTube because of this sick and ugly thing.
But now we must ask ourselves: is that video CP?
No, for everyone else.
Yes for these sick bastards.
How do you classify that?
If it is classified, what happens to the parents when they switch from Android to iPhone and backup the photos and videos to "summer memories 2017" in iCloud?
The definitions are horrifyingly, depressingly, tragically very clear. I did not enjoy reading them.
I think we more or less agree.
While I fully admit that NCMEC could do a better job with transparency and auditing, they are currently being used by several other platforms right now (Facebook, Google, Microsoft) without issue.
Could bad actors inject hashes of non-CSAM content into the database somehow? Well even if they could do this, Apple employes human reviewers who must visually confirm that the flagged photo contains actual CSAM before they report the image. If the image does not contain CSAM, Apple is under no legal obligation to report it.
More information here: https://twitter.com/AlexMartin/status/1424703642913935374/ph...
You’re telling me that you believe a low-wage worker reviewing the worst of the worst human depravity, is going to stand up on a soap box and defend another nameless and faceless denizen of the Earth, when the crux of the argument is basically this:
“Yeah I know the person tripped the safety threshold for CSAM, but these images aren’t that bad!”
It’s not reasonable to trust the human reviewer. They put themselves at risk by going against the automated system. I’d bet 95% of people in this circumstance would pass the buck to the FBI to make their determination, at which point “your” life is already ruined.
A) Images which are a correct hash match to an image already known to NCMEC or other agencies which have already been assigned CSAM category A1 (A=prepubescent, B=sex act);
B) Images which are a hash collision. According to Apple, the likelihood of a collision is 1 in 1 trillion per user account.
This system isn't a child detector strapped to a porn detector, being backed up by a low-wage worker making legal or editorial judgement calls. It's searching for images already known to child safety organisations—and even then only the most unambiguously horrific classification within the set of known images, far far far beyond the point where any ambiguity could possibly reside.
You know that a perceptual hash has more in common with a classifier than with a cryptographic hash, right?
If they were using a crypto-hash, then, yes, your argument could be valid, but with perceptual hashing your picture of your baby in the bath could VERY well generate the same hash as a CSAM image. And nobody believes Apples "1 in a trillion" number.
Things may work right now in 2021. Things will always be changing. New hashes will be introduced. New code will be introduce. New laws in different countries will be introduced.
Now that Apple has introduced this technology that no other phone manufacturer has, it can and will be changed to decrease privacy and increase government control. If China passes a law that states that IPhone needs to scan everyone’s phones for anti-government material, do you really think Tim Cook will say no? They already acquiesced about storing iCloud backups unencrypted on Chinese servers.
And before you say that China would never do that, China and Hong Kong are hunting down the people who were videoed booing Chinese anthem in a shopping mall.
Speak for yourself, because that didn't surprise me at all. When your corpus of "copyrighted" material is so utterly massive and almost entirely devoid of defined rules or boundaries, this kind of error is inevitable. If anything I'm more surprised that we haven't seen even more of these kinds of matches, like the sound of generic telephone ringtones, recordings of mechanised church bells, or machinery which performs highly uniform tasks, etc.
In the case of A1-categorised CSAM, the quantity of items is many orders of magnitude lower, the degree of technical curation will be orders of magnitude higher, and the thresholds for matches will be narrower. Is there a chance that the A1 corpus will have a few images that should have been classified as A2, B1 or B2? Yes. Is there a chance that it includes pictures of The Statue of Liberty or Westminster Tower? Almost certainly not.
China can do whatever China wants. If China wanted Apple to scan the iPhones of Chinese residents for pictures of Winnie The Pooh, they could have done that last year. They pass a law, Apple must comply or leave. They don't have a choice.
Of course I don't like it. I think many things China does are awful. But at the end of the day I wouldn't stand for China exporting their morality onto me, and I'm not a hypocrite.
They aren't doing simple hash matching, they're doing fuzzy matches on the hashes, so there will be far more false positives than just hash collisions.
Apple started out saying that they can't decrypt data from anyone's phone. They fought a lawsuit from the FBI over the San Bernadino terrorist phones. This is one of the reasons why I went all-in on Apple, because they were willing to fight the government over our privacy.
Now, years later, they don't encrypt iCloud backups because the FBI told them not to.
Google used to human review all copyright violations on Youtube. Fast forward a few years later, and all copyright violations are demonstrably shown to be approved and the content generator needs to prove that they didn't violate copyright. Look at the violations over white noice. Google doesn't even care anymore, they just let the copyright violations go through and affect the content creators with no review.
To believe that Apple employees will review CSAM-flagged photos 5 years from now is so incredibly naive, it's actually funny. You can bet they are working on AI that will handle this for them right now in Cupertino.
And then, it will be random chance whether or not we are flagged and labelled as pedophiles, or if a government wants to tag us as "problematic" and wants access to our phones because we are journalists and they want to see our anonymous sources.
It's naive to think that it won't go in this direction.
If you're a pedophile, after this announcement you will delete all the photos off your iPhone and never use it again. After the first few rounds of arrests and cleansing, it will be well knowing within the pedophile community not to use iPhones. And then the only ones who will be getting their photos scanned will be innocent people. So the entire feature doesn't make sense at all. It's a ruse.
You have to first get a “significant” number of photos flagged, then they have to pass Apple’s manual review (ie looking at photo thumbnails), and only then does Apple report the account.
Again I’m not in favor of this system but I think a lot of the criticism misses the mark.
This is a step too far (on-device vs in cloud) but all of the “but what if the govt…” is ridiculous because that happens anyway if the govt wants it.
Maybe we should change the government.
This "much more simply" mechanism is exactly what was created by Apple. It's done right now, and there's no need to worry about how much more simply they COULD HAVE.
It's a fait-accompli. The government wanted it, and now they have it. Now they can scan individual phones for whatever they want.
Yes the government could order them to change the system. They could also order Apple to create the system in the first place without all the indirection, safety vouchers, human review, etc which make it inefficient as a direct surveillance tool.
They could simply tell Apple that this is for CSAM so that people would support it, and then change it later after everyone is is acclimatized to it and forgot about it.
Which is exactly what had happened. And exactly what will happen in a few years.
You sound like a government plant trying to gas light people into thinking this isn’t a big deal. It is a huge deal and it’s not as simple as saying “the government could have asked for it much simpler!” This is them asking for it, plain and simple, and Apple delivering on it.
But no, I don’t believe that a government could “fool” Apple by adding non-CSAM images to the database. The review step would catch that.
I don’t like on device scanning in principle and in precedent. I’m just saying this specific tech stack doesn’t seem like it would be useful for your surveillance scenario, and most of your criticisms don’t seem to be based in having read how this system actually works.
I’m AGAINST this system, I just wish the discussion here weren’t so full of misinformation and bad assumptions.
How long until they try to machine-learn based on that database? The door's open.
I no longer trust Apple, and I’m going to get rid of my iPhone.
The CCP already gets Apple to censor the Taiwanese flag emoji and store all Chinese iCloud user data on CCP-accessible servers in China.
We know Apple presently actively and eagerly cooperates with CCP censors to be permitted to operate and sell in China.
This is tailor-made for CCP abuse.
1) During the course of investigation an officer infiltrates a CSAM sharing ring and/or poses as a customer for CSAM. Material is shared with the officer as it would be to an actual consumer of CSAM.
2) When someone is charged with child abuse, possession of child porn, etc, their physical and electronic lives will be methodically and forensically searched for CSAM material. They will likely find material they already know about, but potentially uncover new material and/or new social networks.
Any material acquired would need to be analysed and classified for the purpose of effective prosecution. My understanding is (from other comments made by people on other websites) that images in the NCMEC database are tagged based on the severity of their content and that Apple is only scanning for the most extreme "A1" material.
I wasn't sure what A1 meant so I googled it. According to this[0] PowerPoint presentation, page 22:
A = prepubescent minor
B = pubescent minor
1 = sex act
2 = "lascivious exhibition"
[0] https://www.prosecutingattorneys.org/wp-content/uploads/Pres...
"The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. This is further mitigated by a manual review process wherein Apple reviews each report to confirm there is a match..."
So no, "one in a trillion" doesn't include manual review.
> We want to ensure that the reports that we make to NCMEC are high-value and actionable, and one of the notions of all systems is that there’s some uncertainty built in to whether or not that image matched. And so the threshold allows us to reach that point where we expect a false reporting rate for review of one in 1 trillion accounts per year.
https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
Leaves some ambiguity but it sounds like the reporting to gov is 1/1t
The other side started it first. Look at the government's dubious claims about terrorism prevention. John Walsh a founder of NCMEC testified to congress that millions of children were abducted every year and that america was "littered with mutilated, decapitated, raped, strangled children," (this was and is not true).
If fear mongering about Big Brother throwing normal people in prison for pictures of their children is what it takes to blunt the expansion of the surveillance state I say fair play.
True, but the wording of that condition was very vague... the threshold could be 1.
like most privacy invasions these days, its casting a widenet to put a dent in a problem that sill inevitable just route around it, and soon enough itll just be turned into a copyright cashgrab
What are you talking about? Ask anyone who has worked in this space: false positives are abound[1][2], especially when you're looking for fuzzy matches. And you have to look for fuzzy matches otherwise slight modifications to illegal images would bypass the detection system.
> Photos of your own kids won't trigger it, nude photos of adults won't trigger it;
This is also incorrect. In general, if two images kind of look like one another when you squint, they're going to have similar perceptual hashes. A lot of unrelated things look similar to one another when you squint, and a lot of unrelated things are going to have similar perceptual hashes. And, again, you'll be doing fuzzy matches on these hashes, so you're going to pick up those unrelated things even more so than when you just have hash collisions.
Even if the tech has 1 in a trillion chance, it will happen a lot with billions of people generatin thousands of ilage every years. And of course, the hash database being abused.
I assume they are just standard ML.
>"...report iCloud users who store known Child Sexual Abuse Material (CSAM) in their iCloud Photos accounts."
OK. They're not trying to tag and bag your images for 'abuse content'.
If you collect your child abuse porn on iCloud, we're going to report you?
For everyone else getting access to the phone and manually loading the pic would work.
Additionally it's possible to fool AI: https://slazebni.cs.illinois.edu/fall18/lec12_adversarial.pd...
Be aware that almost all cloud providers screen photos. Facebook reported 20 million images in 2020, Google reported half a million. Dropbox, Box, and many, many others report images. See https://www.missingkids.org/content/dam/missingkids/gethelp/... to see a complete list of companies that screen and report images.
The other thing Apple announced which is completely separate from the CSAM photo scanning is additional parental controls for the Messages app. If a parent opts in for their under-13 children, a machine learning model will look for inappropriate material and warn the child prior to showing the image. The child is also told that their parent will be flagged if the child looks at it anyway. For 13-18 year olds whose parents opted in, the teen is warned first about the content. If the teen continues past the warning the image is shown and no further action is taken. Parents are not flagged for children 13 and over. As I said, this is a parental control for pre-adult kids. It requires opt-in from the parents and has no law enforcement implications.
Can you imagine the chaos of a successful collision matching some explicit material being sent as a prank or targeted attack?
And in the example you gave we are talking about Google, not some early-stage understaffed startup.
1. Who is adding these photos to NCMEC? 2. How often are these photos added? 3. How many people have access to these photos - both adding and viewing?
Everyone is focused on Apple and no one is looking at MCMEC. If I wanted to plant a Trojan horse, I would point everyone towards Apple and perform all of the dirty work on the NCMEC end of things.
This initiative makes me extremely leery of black boxes, to the extent that any algorithm between subject and accusation had damned well better be explainable outside the algorithm; else I as a jury member am bound to render a "not guilty" verdict.
We don’t know really how adding new hashes work. NCMEC has the whole new algorithm and they drag-n-drop new images? Hopefully not like that.
There are legitimate things to be concerned about, but 99% of internet discussion on this topic is junk.
There’s also the Op-Ed by Matthew Green and Alex Stamos, cyber security researchers: https://www.nytimes.com/2021/08/11/opinion/apple-iphones-pri...
I think John's article is better than Ben's, but they're both worth reading.
Ben takes the view that unencrypted cloud is the better tradeoff - I'm not sure I agree. I'd rather have my stuff e2ee in the cloud. If the legal requirements around CSAM are the blocker then Apple's approach may be a way to thread the needle to get the best of both worlds.
I think I'd rather have the non-e2ee cloud.
Let the devil in, and he'll treat himself to tea and biscuits.
The only thing sent is the hash and signature and that's only if there are enough matches to pass some threshold.
I don't really view that as 'permanently compromised' - at least not in any way more serious that Apple's current capabilities to compromise a device.
I think e2ee still has meaning here - it'd prevent Apple from being able to see your photo content on their servers.
This is a nuanced issue, I don't think there's an obviously better answer and both outcomes have different risks. [0]
[0]: https://www.lesswrong.com/posts/PeSzc9JTBxhaYRp9b/policy-deb...
Though I'd argue the risk has kind of always lied there given companies can ship updates to phones. You could maybe argue it'd be harder to legally compel them to do so, but I'm not sure there's much to that.
The modern 'megacorp' centralized software and distribution we have is dependent on policy for the most part.
It does have some advantages - it's easier to argue (see: the disaster that is most of the commentary on this issue).
It also could in theory be easier to argue in court. In the San Bernardino case - it's easier for Apple to decline to assist if assisting requires them to build functionality rather than just grant access.
If the hash detection functionality already exists and a government demands Apple use it for something other than CSAM it may be harder for them to refuse since they can no longer make the argument that they can't currently do it (and can't be compelled to build it).
That said - I think this is mostly just policy all the way down.
Given the amount of nuance here, I also think it's important to differentiate between the FBI showing up and asking for something and government passing laws forcing encryption backdoors. The former is what Apple has fought to date b/c they can. The later is much harder to fight and Apple will most likely have to comply regardless of what features already exist or not (see China/iCloud). The later is also the most dangerous since politicians rarely understand technology enough to do something sensible. It remains to be seen, but Apple could be trying to get in front of long term law changes with an alternate solution.
They've turned your device into a dragnet for content the powers that be don't like. It could be anything. They're not telling you. And you're blindly trusting them to have your interests at heart, to never change their promise. You don't even know these people.
You seriously want to cuddle up with that?
They're pretty explicitly telling us what it's for and what it's not for.
> "And you're blindly trusting them to have your interests at heart, to never change their promise. You don't even know these people."
You should probably get to work building your own phone, along with your own fab, telecoms, networks, - basically the entire stack. There's trust and policy all over the place. In a society with rule of law we depend on it. You think your phone couldn't be owned if you were important enough to be targeted?
Nobody should blindly trust Apple. As an organization, they already love secrecy and shadows--what better place to sneak in and test this kind of feature, free from employee ethics and scrutiny?
They've been cooking this up without telling anyone, which is also indicative of how above board they are. Who knows what else they're doing with this now or will do in the future.
The CIA, FBI, MI6, Mossad, FSB, CCP, et al. will use this to learn more about their targets.
This is what you sound like. The problem here isn't the tech. It's that Big Tech has deluded society into believing privacy and personal ownership of devices doesn't exist because it would inconvenice Big Tech. Law enforcement echoes it because they were spoiled by the brief period that they tasted ClearNet, and they don't want to return to having to investigate the old fashioned way.
Every other major industry has increasingly started doing the same thing. It is not okay. We have no right to sell out future generation's privacy. It's cowardly, selfish, and does more harm to them in the long run.
We’re trusting a lot of the stack. Apple’s policy as described is reasonable. If you distrust it because of the things they could do, that same logic applies to the entire stack.
The nature of modern software distribution is that the majority of the stuff we use from centralized corporations is governed by policy, not technical capability or controls, and you don’t get to know the details.
You don’t own the OS you use, you don’t own the important parts of your phone.
This can be different. Decentralized applications via protocols are interesting (DeFi blockchain stuff like Audius or other apps on Ethereum). If the UX can get figured out.
Urbit is interesting too - if you want to actually own your stack, use Urbit: https://media.urbit.org/whitepaper.pdf
Outside of decentralized protocols you’re ultimately just trusting policy somewhere. It seems dumb to me to arbitrarily be upset at Apple’s policy here, when the specifics are reasonable (and allow for e2ee).
Apple is performing warrantless searches, and somehow people see it as okay because Apple is not the government (even though it functions as a state agency in this regard).
Not true. If there are enough matches, someone at Apple will have a look at your pictures. Even if they are innocent.
That's if there is enough matches to trigger the threshold in the first place, otherwise nothing is sent (even if there are matches below that threshold).
Alternatively this is running on all unencrypted photos you have in iCloud and all matches are known immediately. Is that preferable?
So it is sending pictures? That makes your argument quite a bit weaker.
> Is that preferable?
Nope, E2EE without compromises is preferable.
> Nope, E2EE without compromises is preferable.
Well that's not an option on offer and even that has real tradeoffs - it would result in less CSAM getting detected. Maybe you think that's the acceptable tradeoff, but unless government legislatures also think so it doesn't really matter.
This isn't the clipper chip, this is more about enabling more security and more encryption by default but still handling CSAM.
The CSAM issue is a real problem: https://www.nytimes.com/interactive/2019/09/28/us/child-sex-...
It should and can be an option. Who cares what they offer us. Do it yourself.
If you do it yourself none of this policy stuff matters.
Important to note this is only ran on images going to iCloud so they are already sent.
They are now scanning on the device. Regardless of how limited it is in its current capabilities, those capabilities are only prevented from being expanded by Apple's current policies. The policies enacted by the next incoming exec who isn't beholden to the promises of the previous can easily erode whatever 'guarantees' we've been given when they're being pressured for KPIs or impact or government requests or promotion season or whatever. This has happened time and again. It's been documented.
I really am at a loss how you can even attempt to be fair to Apple. This is a black and white issue. They need to keep scanning for crimes off our devices.
So to your answer your question, yes it is preferable to have them be able to scan all of the unencrypted photos on iCloud. We can encrypt things beforehand if need be. It is lunacy to have crime detecting software on the device in any fashion because it opens up the possibility for them to do more. The people in positions to ask for these things always want more information, more control. Always.
The above reads like conspiracy theory but over the past couple of decades it has been proven correct. It's honestly infuriating to see people defend what's going on in any way shape or form.
This is a policy issue in both cases - policy can change (for the worse) in both cases.
The comparison is about unencrypted photos in iCloud or this other method that reveals less user information by running some parts of it client side (only if iCloud photos are enabled) and could allow for e2e encryption on the server.
The argument of "but they could change it to be worse!" applies to any implementation and any policy. That's why the specifics matter imo. Apple controls the OS and distribution, governments control the legislation (which is hopefully correlated with the public interest). The existing 'megacorp' model doesn't have a non-policy defense to this kind of thing so it's always an argument about policy. In this specific implementation I think the policy is fine. That may not hold if they try to use it for something else (at which point it's worth fighting against whatever that bad policy is).
Apple's good solutions to the CSAM problem (which I think thread the needle for a decent compromise) could prevent worse policy from the government later (attempts to ban encryption or require key escrow like in the 90s).
Basically what I said here: https://news.ycombinator.com/item?id=28162418
This implementation as it stands reveals less information about end users and could allow them to enable e2ee for photos on their servers - that's a better outcome than the current state (imo).
1. Encrypt everything in the cloud but upload the hashes of these items as well on the device. Also notify us so we can notify law enforcement if they're doing some illegal stuff.
2. Everything is unencrypted in the cloud. No actions are taken on the device. No notifications to authorities.
With option one the sanctity of the device ownership is breached. With option two it's maintained. Maintaining that stark distinction is hugely important for what future actions can be taken in the public eye. Normalizing on device actions that work against the user must be fought at every instance they occur.
Your line of thinking dangerous because you're ignoring the public perception of a device you own actively working against you. Apple's behavior cannot be allowed to be considered normal.
Option #1 doesn’t really breach the ‘sanctity of device ownership’ because it only occurs if you’ve enabled iCloud photos on the device.
Option #1 seems better to me in its current implementation. I understand the fear of abusing the hash matching. I just think that’s a separate thing.
I'm done. You'll rationalize anything.
I don't think it's a rationalization to point out that it only occurs when the same baseline conditions are met (using the cloud). I think those constraints/specifics matter. I wouldn't be in favor of the policy if they were different (and I'm not even sure I'm in favor of it now).
My personally preferred outcome would be e2ee by default for everything without any of this, but I also understand the concerns of NCMEC and the general tradeoffs/laws around this stuff (and future regulatory risk of CSAM) - and just the general issue of reducing child sexual abuse.
I am also in favour of E2E by default for everything without any device or cloud based scanning. However, Apple doesn't want to be caught in having developed a service that enables for child exploitation. Doing nothing may have even more invasive requirements legally forced by government, so Apple is stuck with a dilemma. Also lets not forget that Apple should also not want child exploitation to occur and therefore also should do something.
The question I have for drenvuk is how else is Apple able to prevent or detect child exploitation and the storage or distribution of content such as this on Apple's services?
I suspect they’re trying to get ahead of that and solve this in the most privacy protecting way possible.
No, but at a "visual derivative"
If uploading a perceptual hash of a photo breaks 'local vs cloud separation', the uploading the whole photo in the clear surely does the same.
I like that they disagree - the issue doesn't have an obviously correct answer.
Seems like the existence of this scanning agent by default makes it not E2EE anymore
Linked articles and comments have said apple's brand is now destroyed, that apple is committing child porn felonies somehow with this (the logical jumps and twisting to get to these claims are very far from strong plausible interpretation).
How do you scan for CASM in an E2EE system is the basic question Apple seems to be trying to solve for.
I'd be more worried about the encrypted hash DB being unlockable - is it clear this DOES NOT have anything that could be recreated into an image? I'd actually prefer NOT to have E2EE and have apple scan stuff server side, and keep DB there.
> The laws related to CSAM are very explicit. 18 U.S. Code § 2252 states that knowingly transferring CSAM material is a felony. (The only exception, in 2258A, is when it is reported to NCMEC.) In this case, Apple has a very strong reason to believe they are transferring CSAM material, and they are sending it to Apple -- not NCMEC.
> It does not matter that Apple will then check it and forward it to NCMEC. 18 U.S.C. § 2258A is specific: the data can only be sent to NCMEC. (With 2258A, it is illegal for a service provider to turn over CP photos to the police or the FBI; you can only send it to NCMEC. Then NCMEC will contact the police or FBI.) What Apple has detailed is the intentional distribution (to Apple), collection (at Apple), and access (viewing at Apple) of material that they strongly have reason to believe is CSAM. As it was explained to me by my attorney, that is a felony.
Apple is going to commit child porn felonies according to US law this way. This claim seems actually quite irrefutable.
It would be interesting to hear what a court has to say if a child porn consumer would try to defend him/her with this "argument".
Secondly, any derivatives that are clear enough to enable a definitive judgment whether something's CP or not by an Apple employee would be subject to my argument above. Also just collecting such material is an felony.
I don't see any way around that. Only that promising some checks before stuff gets reported for real is just a PR move to smoothen the first wave of pushback. PR promises aren't truly binding…
They can easily say afterwards that they're "frankly" required to directly report any suspicion to enforcement agencies because "that's the law", and they didn't know because that was an oversight?
That would be just an usual PR strategy to "sell" something people don't like: Selling it in small batches works best. (If the batches are small enough people often don't even realize how the whole picture looks. Salami tactics are tried tool for something like that; used for example in politics day to day).
This is sort of what I mean and a perfect example.
People imagine that apple hasn't talked to the actual folks in charge NCMEC.
People seem to imagine apple doesn't have lawyers?
People go to the most sensationalist least good faith conclusion.
Most mod systems at scale are using similar approaches. Facebook is doing 10's of MILLIONS of images to NCMEC, these get flagged by users and/or systems, and in most cases then facebook copies, checks through moderation queue and submits to NEC.
Reddit uses the sexualization of minors flags. In almost all cases, even though folks may have strong reasons to believe some of this flagged content is CSAM, it still gets a manual look. Once they know they act appropriately.
So the logic of this claim about apples late to party arrival of CSAM scanning is weird.
We are going to find out that instead of trying to charge apple with some kind of child porn charges, NCMEC and politicians are going to be THANKING apple, and may start requiring others with E2EE ideas to follow a similar approach.
The one odd thing I don't get. It would be a lot EASIER to just scan everything when its in the cloud itself.
Why go to this trouble to avoid looking at users photos in the cloud, set these thresholds etc. You'd only need to scan on device if for some reason you blocked your own ability to scan in cloud (ie, for E2E photos - which I don't think users actually want).
Facebook checks for potential CSAM when you upload from your client device (sometimes an iphone) to their servers or after it's on their system and a user flags it.
Instagram also check once you upload.
These are all transmissions.
Apple checks if you upload. If you don't upload or attempt to upload to their servers, no check.
All these are to flag potential CSAM. Some do more - nudity in general, harmful content filters etc. Some is auto blocked, some is forwarded for review and report etc.
In almost all cases flagging is part of or connected to uploading to a third parties servers. The flagging for CSAM is not a conviction - some do and some don't do a human review before submission. Most situations where folks can use flagging to hide content get a human review at some level to avoid abuse of the flagging system itself.
Thats not the issue according to the linked source.
Instagram transmits all photos and assumes it's not CSAM until flagged - thats safe content. Apple transmits ONLY CSAM - thats a no-no content because they're assuming it is CSAM and you can't transmit CSAM.
You can't knowingly transmit CSAM. Transmitting a photo pre-scan is safe (if you assume any photo is not csam), transmitting post-scan is dangerous if you filter for csam.
Apple uploads all photos to iCloud photos (just as google does). This includes CSAM and not CSAM.
It keeps a counter going of how much stuff is getting flagged as possible CSAM. They don't even get an alert about anything until you hit some thresholds etc. And no one has reviewed anything yet at all, the system is flagging things up as other systems do.
Are you sure (legally) that one can't review flags from a moderation system? That is routine currently. No one is knowingly doing anything. Their system discusses being alerted to possible counts of CSAM.
Is your goal that apple go straight to child porn reports automatically with no human involvement at all? At scale (billions of photos) that's going to be a fair number of false positives with potentially very serious consequences.
The current approach is that images are flagged in various ways, folks look at them (yes, a large number have problems), then next steps are taken. But the flags are treated as possible CSAM.
Please look into all the false positives in youtube screening before you jump from a flag => sure thing. These databases and systems are not perfect.
> Is your goal that apple
I'm not a lawyer and i want apple to do nothing especially not scan my device.
I'm saying the linked article in discussion says you can't transmit content you KNOW (or suspect) is CSAM. You don't assume that all your customers' content is CSAM, but post-scan, you should assume.
The only legal way to transmit (according to article) is if it's to the government authorities.
I don't know the legal view on the "false positive" suspicion vs legality of transmitting. That's a gamble it seems. I don't have a further opinion on it since IANAL and this is very legal grey area.
A strong claim (apple committing CSAM felonies) should be supported by reasonably strong support.
Here we have a blog post where they've talked to someone ELSE who (anonymously) has reached some type of legal conclusion. If you follow the QAnon claims in this area (there are lots) they follow a somewhat similar approach - someone heard from someone that something someone did is X crime. It's a weak basis for these legal conclusions.
I don't think so.
Apple transfers the images to iCloud, yes, but before the threshold of flagged photos is reached, Apple doesn't know that there might be CSAM material among them. When the threshold is exceeded and Apple learns about the potential of CSAM, the images have been transferred already. But then Apple does not transfer them any further, and has a human review not the images themselves, but a "visual derivative" that was in a secure envelope (that can by construction only be unlocked once the threshold is exceeded).
If the author were correct Facebook and others would have been charged with felonies already. Verification happens in good faith and requires transmission despite the exact letter of the law. The people doing it regularly talk to NCMEC. There are evidentiary and chain of custody procedures to follow that also transmit the material. Guess we should charge computers with felonies!
It’s also fucking hysterical that the armchairs who didn’t know what CSAM stood for a week ago think that one of the most litigiously sensitive systems ever built by a corporation somehow missed criminal legal liability. You know, it’s pretty easy to overlook criminal liability when you’re building a system that is used to establish criminal liability. Totally passes the smell test. Irrefutable indeed.
You guys are off your rockers and should move on to another topic. Seriously. Every day this is discussed is another harsh reminder of (a) how few fucks the industry gives about abuse of children and (b) how everyone here digests blogs and considers themselves authoritative on horrors they’ve never once experienced. Every day this is argued on HN, particularly last night when someone said the privacy situation is “the actual abuse” and “much worse” than the rape of children, is another day I’m ashamed to have chosen this profession and work alongside you. This community welcomes the people who have built surveillance systems for every consumer activity imaginable and kids getting molested is where you draw the privacy line, huh? Can’t look for that? I’m genuinely out. Keep your industry. I’d rather manage a Taco Bell if this is the perspective of this industry, because brother, I’ve seen what they’re trying to tackle and I’m still in therapy.
There is a hypocrisy at the core of the “privacy” argument here that is fundamentally indicting not only this community, but everyone discussing this up to and including EFF. I’ve never been so disappointed in people who I used to look up to and think of as good, smart folks.
Source: I’ve built CSAM handling systems for a FAANG and written the policy for using it. The author has misinterpreted the very law he cites and overlooked two subparagraphs. But that isn’t what you want to hear.
You WILL be downvoted.
I'm a parent, it's also crazy to me how THIS of all things is where folks are going crazy over privacy. Literally, they will build something to track your every mouse move, scan every photo, log and sell all your browsing history and TV watching history (including big ISPs and mfgs).
And this is the thing folks get outraged about? Apple can already scan your stuff on their servers (and should!).
Instead of apple being charged criminally, other companies that do any kind of E2E without this may be required to do something like this. That's my prediction.
We will see if these "irrefutable" claims amount to anything like a child porn charge against apple.
It's crazy to me how you fail to see that instead of everyone playing cyber cowboys and child predator indians the focus for preventing child abuse should be in real life. Criminalizing content does absolutely nothing for those kids who are abused by someone close to them. And unfortunately the vast majority (~75%) of cases happen like that.
If the stats are even remotely right it's unfathomable how bad prevention and deterrence is. (Over the course of their lifetime, 28% of U.S. youth ages 14 to 17 had been sexually victimized -- https://victimsofcrime.org/child-sexual-abuse-statistics/ )
This shows how bad our aggregated priorities are. War on X so far only made X worse. ¯\_(ツ)_/¯
I don't doubt for a minute that victims continue to suffer from the knowledge that there's a lot of traumatizing content on the Internet, and that various services integrating with clearinghouses help.
I also don't doubt that Apple is at least semi-competent and could pull this off in an okay-ish way. But all the goodwill and clout that these megacorps have would have been better spent on advocating for policies that prevent child abuse. (Neglect, physical and sexual.)
https://www.law.cornell.edu/uscode/text/18/part-I/chapter-11...
Instead of exclusively focusing on the authoritarian slippery slope like it's inevitable, it's worth wondering first: why do the major tech companies show no intention of giving up the server-side PhotoDNA scanning that has already existed for over a decade? CSAM is still considered illegal by half of all the countries in the entire world, for reasons many consider justifiable.
The point of all the detection is so that Apple isn't found liable for hosting CSAM and consequently implicated with financial and legal consequences themselves. And beyond just the realm of law, it's reputational suicide to be denounced as a "safe haven for pedophiles" if it's not possible for law enforcement to tell if CSAM is being stored on third-party servers. Apple was not the best actor to look towards if absolute privacy was one's goal to begin with, because the requests of law enforcement are both reasonable enough to the public and intertwined with regulation from the higher powers anyway. It's the nature of public sentiment surrounding this issue.
Because a third party insisting that user-hosted content is completely impervious to outside actors also means that it is possible for users to hide CSAM from law enforcement using the same service, thus making the service criminally liable for damages under many legal jurisdictions, I was surprised that this debate didn't happen earlier (to the extent it's taking place, at least). The two principles seem fundamentally incompatible.
My own guess is that the encryption is there so that people won't have access to an up-to-date database to test against. People who want to intentionally create false positive could abuse it, and sites that distribute images could alter images to automatic bypass the check. There is also always the "risk" that some security research may look at the database and find false positives from the original source and make bad press, as they have done with block lists (who can forget the bonsai tree website that got classified as child porn).
People are furious with Apple, and there's no reason to discount the completely legitimate concerns they have. This is a slippery slope into hell.
It's a good thing congress is about to start regulating Apple and Google. Maybe our devices can get back to being devices instead of spy tools, chess moves, and protection rackets.
(read: Our devices are supposed to be property. Property is something we fully own that behaves the way we want. It doesn't spy on us. Property is something we can repair. And it certainly is not a machination to fleece the industry by stuffing us into walled and taxed fiefdoms, taking away our control. Discard anything that doesn't behave like property.)
[edit: I've read Gruber's piece on this. It's wish-washy, kind of like watching a moderate politician dance on the party line. Not the direct condemnation this behavior deserves. Let's not take his wait and see approach with Dracula.]
this is not strong safety for citizens
source: political history
You mean the same Gruber who described the situation as “justifiably, receiving intense scrutiny from privacy advocates.”? The one who said “this slippery-slope argument is a legitimate concern”?
I'm having a hard time reconciling your pat dismissal with the conclusion of his piece which very clearly rejects the position you're attributing to him as grounds for dismissal:
> But the “if” in “if these features work as described and only as described” is the rub. That “if” is the whole ballgame. If you discard alarmism from critics of this initiative who clearly do not understand how the features work, you’re still left with completely legitimate concerns from trustworthy experts about how the features could be abused or misused in the future.
I mean, sure, know where he's coming from but be careful not to let your own loyalties cause you to make a bad-faith interpretation of a nuanced position on a complex issue.
However, if we consider the slipery slope, under pressure from a shaddow government, the contents of your phone could have been uploaded to the CIA every day, including live recordings 24 hours a day.
Sure, a suitably powerful authoritarian org could do lots of secret things, but that isn't what happens in real life: in real life you publicly change policy in increments and get everyone to go along with it.
"Apple was actually scanning all users photos for CSAM regardless of iCloud usage due to a bug in the most recent firmware" is a headline that is guaranteed in the future.
Care to put a date on your guarantee?
You’re describing new functionality which would have to be added in many places: in addition to building that service they have to turn off the recording indicators and prompts, coexist with other apps recording, not have recording pause playback in other apps like normal, masking data usage on both the phone and your carrier’s reports, concealing the battery loss and putting in bigger hardware batteries to compensate, etc.
That’s technically possible but there’s no link to this feature - that hypothetical government would need to do the same things either way. It’s similarly not Apple-specific: with that level of control the same thing would happen Android and anything else.
Phone has functionality to backup its contents. Phone has functinality to record things. No new functionality needed.
Thus, they are backdoors built into the system with the ability to record everything and upload it to an authourtarian regime for the genocide of the human race.
But we all know the real evil here is using a hashing algorithm to check images you upload to their server for known kiddie porn.
Given the political/societal climate, it probably gets them more donations.
Ah, I see you work for Apple, given the language ("the screeching voices of the minority").
Two different things which both are known to be prone to all kind of miss-detection.
So, three different things.
I don't know how much you know about them, but this is what the EFF's role is. Privacy can't be curtailed uncritically or unchecked. We don't have a way to guarantee that Apple won't change how this works in the future, that it will never be compromised domestically or internationally, or that children and families won't be harmed by it.
It's an unauditable black box that places one of the highest, most damaging penalties in the US legal system against a bet that it's a perfect system. Working backwards from that, it's easy to see how anything that assumes its own perfection is an impossible barrier for individuals, akin to YouTube's incontestable automated bans. Best case, maybe you lose access to all of your Apple services for life. Worst case, what, your life?
When you take a picture of your penis to send to your doctor and it accidentally syncs to iCloud and trips the CSAM alarms, will you get a warning before police appear? Will there be a whitelist to allow certain people to "opt-out for (national) security reasons" that regular people won't have access to or be able to confirm? How can we know this won't be used against journalists and opponents of those in power, like every other invasive system that purports to provide "authorized governments with technology that helps them combat terror and crime[1]".
Someone's being dumb here, and it's probably the ones who believe that fruit can only be good for them.
You would have to have not one, but N perceptual hash collisions with existing CSAM (where N is chosen such that the overall probability of that happening is vanishingly small). Then, there'd be human review. But no, presumably there won't be a warning.
> Will there be a whitelist to allow certain people to "opt-out for (national) security reasons" that regular people won't have access to or be able to confirm?
Everyone can opt out (for now at least) by disabling iCloud syncing. (You could sync to another cloud service, but chances are that then they're scanned there.)
Beyond that, it would be good if Apple built it verifiably identically across jurisdictions. (If you think that Apple creates malicious iOS updates targeting specific people, then you have more to worry about than this new feature.)
> How can we know this won't be used against journalists and opponents of those in power, like every other invasive system that purports to provide "authorized governments with technology that helps them combat terror and crime[1]".
By ensuring that a) the used hash database is verifiably identical across jurisdictions, and b) notifications go only to that US NGO. Would be nice if Apple could open source that part of the iOS, but unless one could somehow verify that that's what's running on the device, I don't see how that would alleviate the concerns.
It really doesn't matter how they do it now that we know that iOS has vulnerabilities that allow remote monitoring and control of someone's device to the extent that it created a market for at least one espionage tool that has lead to the deaths of innocent people.
I remember when the popular way to shut down small forums, business competitors, or get embarrassing information taken off the web was to anonymously upload CP to it and then report it, repeatedly. With this, what's to stop virtual "SWATing" of Apple customers? Not necessarily just those whose Apple products have been compromised, whose iClouds have been compromised, or who are the victims of hash collisions (see any group of non-CSAM images that CSAM detection flags).
Will Apple analyze all hardware to ensure no innocent person is framed because of an undisclosed vulnerability? What checks are being offered on this notoriously burdensome process on the accused?
>If you think that Apple creates malicious iOS updates targeting specific people, then you have more to worry about than this new feature
Oof. Good point.
EFF is an advocacy group, and you need to read between the lines what they say because they have a specific set of principles that may or may not align with reality. They published a bad article and took an extreme stance about what could be as opposed to what is.
Parents care about children sending or receiving explicit material. This is for behavioral, moral and liability reasons.
When your 12 year old boy sends a dick pic to his girlfriend, that may be a felony. When your 16 year old daughter sends an explicit picture to her 18 year old crush, that person may be committing a felony by possessing it.
There seem to be an endless number of ways to achieve what they claim without invasively scanning people's private data.
Figure it out.
it happens all the time
A lot of innocent people are going to get caught up in this.
I think some people think that’s the probability of a picture being incorrectly flagged, which would be more concerning given the 1.5 trillion images created in the US.
Source: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
"The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. This is further mitigated by a manual review process wherein Apple reviews each report to confirm there is a match..."
So it's 1 in 1 trillion per account PRIOR to manual review in which the odds of error get reduced even further.
Photos in iCloud are unencrypted and Apple checks for CSAM on the unencrypted photos server side, they know of all matches.
OR
Photo hashes are checked client side and only if a certain threshold of matches is passed does Apple get notified at all (at which point there's a sanity check for false positive by a person). This would allow all photos on iCloud to be able to be encrypted e2e.
Both only happen when iCloud photo backup is enabled.
The new method reduces the risk.
False dichotomy. How about they leave people's data alone?
I think their design is making some really smart trade offs, given the needle they are trying to thread. But it shouldn’t exist at all, in my opinion; it’s too juicy a target for authoritarian and supposedly democratic governments to find out how to squeeze Apple into using this for evil.
"The Messages feature is specifically only for children in a shared iCloud family account. If you’re an adult, nothing is changing with regard to any photos you send or receive through Messages. And if you’re a parent with children whom the feature could apply to, you’ll need to explicitly opt in to enable the feature. It will not turn on automatically when your devices are updated to iOS 15."
And how is that?
It seems like the Gruber article follows a common formula for justifying controversial approaches. First, "most of what you hear is junk", then "here's a bunch of technical points everyone gets wrong"(but where the wrongness might not change the basic situation), then go over the non-controversial and then finally go to the controversial parts and give the standard "think of the children" explanation. But if you've cleared away all other discussion of the situation, you might make these apologistics sound like new insight.
Is Apple "scanning people's photos"? Basically yes? They're doing it with signatures but that's how any mass surveillance would work. They promise to do this only with CSAM but they previously promised to not scan your phone's data at all.
The question is whether an ordinary baby photo is likely to collide with the one of the CSAM hashes Apple will be scanning for. I don't think Apple can give a definite no here (Edit: how could give a guarantee that a system that finds any disguised/distorted CSAM won't tag a random baby picture with a similar appearance. And given such collision, the picture might be looked at by Apple and maybe law enforcement).
Separately, Apple does promise only to scan things going to iCloud for now. But their credibility no long appears high given they're suddenly scanning users' photos on the users' own machines.
Edited for clarity.
Cannot guarantee, but by choosing a sufficiently high threshold, you can make the probability of that happening arbitrarily small. And then you have human review.
> And given such collision, the picture might be looked at by Apple and maybe law enforcement
No, not "the picture", but a "visual derivative".
Do you have any idea that means? Because I certainly don't - how could you possibly identify whether an image is CSAM without looking at something which is reasonably the same image?
What is a visual derivative? Take that algorithm and run it over some normal images and show me what they look like.
All of this is being aggressively talked around because everyone knows it's not going to stand up to any reasonable scrutiny (i.e. plenty of big image datasets out there - does Apple's implementation flag on any of those? Who knows - they're not going to refer to anything specific about how they got "1 in a trillion").
No, I don't know what that means. Presumably it is some sort of thumbnail, maybe color inverted or something.
> does Apple's implementation flag on any of those? Who knows - they're not going to refer to anything specific about how they got "1 in a trillion"
I assume they've tested NeuralHash on big datasets of innocuous pictures, and gotten some sort of bound on the probability of false positives p, and then chosen N such that p^N << 10^-12, and furthermore imposed some condition on the "distance" between offending images (to ensure some semblance of independence). At least that's what I'd do after thinking about the problem for a minute.
What's interesting about this faulty argument is that it hinges an assumption that "innocuous pictures" is a well defined space that you can use for testing and get reliable predictions from.
A neural network does classification by drawing a complex curve between one large set and another large set on a high dimensional feature space. The problem is those features can include, often include, incidental things like lighting, subject placement and so-forth. And this often work because your target data set really does uniquely have feature X. So you can get a result that your system can reliably find X but when you go out to the real world, you find those incidental features.
I don't know exactly how the NeuralHash works but I'd presume it has the same fundamental limitations. It has to find images even they've been put through easy filters that are going to change every particular pixel so it's hard to see how it wouldn't find picture A that looking like picture B if you squint.
https://www.hackerfactor.com/blog/index.php?/archives/929-On...
I don't find this unreasonable.
Context often matters more than the nature of the actual content. Police aquire thousands of images with little hope of ever knowing where they originated. If they are collected by pervs, and could be construed as illegal in the hands of pervs, the images become child porn and can be added to the databases.
What Apple announced is a new system for reading the existing hash lists of known CSAM images and doing the comparison on the device as part of the iCloud upload, rather than on the server after upload.
Protocol is rather device specific (while allowing multi-device), so it might not be enough to access or hack iCloud account to access the photos. So, things get complicated.
Did apple actually say photos would be e2ee or are we just assuming?
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
Parents can produce, distribute and sell CSAM of their own kids. That's one of the implications they'd face in court.
It's not as simple as that. Photos in the NCMEC database are tagged based on the severity of their content. The categories are A1, A2, B1, B2. According to this[0] PowerPoint presentation, page 22:
A = prepubescent minor
B = pubescent minor
1 = sex act
2 = "lascivious exhibition"
[0] https://www.prosecutingattorneys.org/wp-content/uploads/Pres...
The definitions are horrifyingly, depressingly, tragically very clear.
If you do choose icloud upload (most do), they were being uploaded already and stored and may be available to law enforcement.
If you do upload to icloud, NOW they will be screened for matches with "known" images in a database, and if you have more than a threshold number of hits, you may be reported. This will happen on device.
Apple will also scan photos in their cloud system as well from what I can tell (though once on device is working less should land in cloud).
Note that it is HIGHLY likely that google photos / facebook / instagram and others will or are already doing similar scanning and reporting. I've heard millions of reports go in a year.
I wonder what the false positive rates are for:
- A random image against the DB of perceptual hashes
- Images of a baby's skin against the DB of perceptual hashes
It seems like the second would necessarily have a higher false positive rate: similar compositions (contains baby's skin) would more likely have similar chunks. Is it just a little higher or several orders of magnitude higher?
I know hash collisions are rare, but wonder how much rarity of collisions decreases with perceptual hashes.
I'd also like to know more about the specifics here, my guess is that threshold value is pretty high (their 'one in a trillion' comment not withstanding). It's probably targeting large CSAM dumps of matches which would not get flagged by different images.
Images then do get a manual review before a report is made which is good and may help provide feedback on alogs being used.
Going to be hard though for apple to set the second factor to high - I'd say 5 maybe? It's hard to say you had matches on potential CASM and ignored them I'd think.
https://rentafounder.com/the-problem-with-perceptual-hashes/
the false-positive rate will be likely high. Given the billions of pictures going through this system there are going to be a lot of false accusations of child porn possession likely (and alone such an accusation can ruin lives).
HN discussion of that article from a few days ago:
Just a minor policy change (lower threshold) away from calling millions of people pedophiles.
Disabling iCloud does not remove the uploading system from your phone.
Pressing end recording on a video does not remove the video capture system from your phone.
on edit: later on of course this will make a great article in some place like the Atlantic with a stolid monochromatic picture of your family in the lead-in and we will all read about it on HN and talk about how this was an obvious problem with the whole system (if it gets posted at the right time and gets enough upvotes).
Current CSAM depends on humans to "verify" the imagery, this is something companies desperately want to get rid of, and so do the employees understandbly so. Nobody wants a job (99.99%) of comparing CSAM. It costs companies money in labor costs, and draws them bad PR when those employees inevitably develop permanent/semi-permanent mental health issues from it.
The only reason it hasn't happened yet is because a startup can't just start scanning CSAM. They need the blessing of the feds to do that, which requires political connections, and of course requires competing with companies that already have that blessing - something that politics prevents.
PhotoDNA and current CSAM scanning only gets known CSAM, but not new CSAM. The end goal is to detect CSAM before it's ever even distributed, to be "closer to the victim", rather than just those consuming it.
Even with current PhotoDNA you can generate hash collisions, which flag the image for review, and a real human compares the material. This is of course subject to change for the reasons stated above.
Secondarily, automatic scanning and ID'ing of imagery is how you can easily throw an FBI raid at someone. Apps like Telegram automatically download every image/video in the thread.
Ontop of that you can create images that appear different at differing resolutions. At one resolution a harmless meme, at another, CSAM. Meaning that you can again throw an FBI raid at someone using simple tricks.
* As addressed in the comments below, this isn’t entirely true: the hash looks for visually similar picture and there may be false positives.
The NCMEC database and this hashing have been around for like 15 years. I’m curious as to how you know this.
In a TechCrunch interview Apple said that they are going after larger targets that are worth NCMEC’s time.
Apple's on-device list of hashes only includes images which have been classified "A1" under the CSAM categorisation scale. If any other photographs are accidental hash collisions to these images, it's going to be pretty damn obvious to the human reviewer.
I just don't have that trust. Obviously I think that protecting children is an incredibly important thing to be doing, but I don't trust apple to be running such a system(I do trust them maybe a tiny bit more than I'd trust google in this case, but ultimately I'd rather this system didn't exist at all).
It's actually worse than this, if the hashes are similar then they'll get sent for review. Your picture could be a picture of an abstract painting[0] which has no visual similarity to anything in the db, but through the magic of crypto is similar and it too will be flagged.
[0] The reason I use this example is because someone posted a bunch of abstract art that was flagged by the algo.
No. If the number of matches to known CSAM in your library exceeds a threshold, then a person will look at a "visual derivative" of only those pictures whose perceptual hatch match that of known CSAM.
Note that, if I understand correctly, pictures that Android users sync to Google have already been scanned for some time. Where are all those false positives?
Apple very specifically said that their employees will look at the suspected picture before sending it through to authorities. Where do you see the bit about visual derivatives? What would that even mean or look like?
Also what is this threshold? As others have pointed out, us parents have literally hundreds of pictures of our newborns, toddlers and kids - having to trigger some detector "multiple" times doesn't give me any peace of mind at all.
>>Where are all those false positives?
Google doesn't use perceptual hashing, or at least haven't said they do.
In Apple's white paper about the proposed feature:
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
See the links at the bottom here for more:
https://www.apple.com/child-safety/
> Also what is this threshold?
The "perceptual hash" is supposed to match a specific image (though possibly cropped, or otherwise altered a bit, such as through a filter), not "toddlers" per se.
> Google doesn't use perceptual hashing, or at least haven't said they do.
I don't know what the other cloud providers are doing, but I'd be very surprised if they use (trivially circumventable) cryptographic hashes.
https://www.hackerfactor.com/blog/index.php?/archives/929-On...
[1] you could "help" independence by requiring a certain distance between images you simultaneously flag.
So if your innocent baby pic looks similar enough to a previously tagged child abuse image then YES, it will flag you and send a copy to the feds.
And before you correct me, the Apple employee will see a picture of your naked baby and hit “forward to NCMEC”, which… upon investigation is actually just the feds
Are news filled with false-positive accusations by PhotoDNA, flagging wrong images in Google, Facebook, Instagram etc.?
No, they are not. Ask anyone who has worked in this space[1][2], including myself. They are incredibly common.
Two images that kind of look like one another will have similar or the same hashes. That is the point of perceptual hashing.
This only catches ownership of illegal photos.
Enjoy explaining why your mugshot and arrest record had these charges attached to it!
(Actually, in this case the prosecution would probably use the other pictures on the phone that were not detected by the scanning tool as a way to get a guilty plea deal!)
But it can be a form of denial of service: saturate the system with hash collisions so that people can't keep up.
LEO's/FBI/every other institution/group that deals with child pornography and abuse have teams that go through a near infinite amount of pictures and videos of CP/etc.
These are then marked by said people as either - yes, CP/Abuse/etc - or marked false positive.
Once marked as what they're after, they're uploaded to a shared database between all groups involved.
Only what is in these worldwide national databases is what's going to be checked against. Your new pictures of your children will have obviously never made their way to any of these groups as they've never been shared/distributed in any areas of the internet/etc these people work in to track down trafficking rings (well, I'd hope you're not selling pictures of your children to them).
This is the way I understand it. I admit I haven't looked into it that much. If it's anything different than what I've said, then yeah, it's probably fucked. I don't get what people don't understand about checking against a database though. No, your new pictures of whatever are not in this pre-existing database
Apple uses two different approaches:
1. Some way to try to detect _known_ child pornographic material, but it's fuzzy and there is no guarantee that it doesn't make mistakes like detecting a flower pot as child porn. But the chance that your photos get "miss detected" as _known_ child pornographic material shouldn't be too high. BUT given how many parents have IPhones it's basically guaranteed to happen from time to time!
2. Some KI child porn detection on child accounts, which is not unlikely to labile such innocent photos as child porn.
The child account iMessage thing is really entirely separate from the CSAM related iCloud announcement. It's unfortunate people keep confusing them.