Critically important? Really?
A system that is trivial to circumvent and which only catches people sharing pics that have already been added to a database is not going to move the needle on actual physical abuse of children.
A system that is trivial to circumvent and which only catches people sharing pics that have already been added to a database is not going to move the needle on actual physical abuse of children.
I think it's important to note that Apple is late to this, and they're just playing catch-up in implementing this scanning. The only difference between Apple and others is that they do it completely (or partially) on-device rather than all on their servers. And they they explicitly told people exactly what and how they're doing it all.
It's kind of a funny situation where in Apple being more transparent than any other implementer of this same process, they've gotten themselves in more strife.
> According to NCMEC, I submitted 608 reports to NCMEC in 2019, and 523 reports in 2020. In those same years, Apple submitted 205 and 265 reports (respectively). It isn't that Apple doesn't receive more picture than my service, or that they don't have more CP than I receive. Rather, it's that they don't seem to notice and therefore, don't report.
> In 2020, FotoForensics received 931,466 pictures and submitted 523 reports to NCMEC; that's 0.056%. During the same year, Facebook submitted 20,307,216 reports to NCMEC.
https://www.hackerfactor.com/blog/index.php?/archives/929-On...
You can argue it's better to have neither type of scanning, but if Apple considers it a critical business issue that their cloud is the preferred hosting site of CSAM[0], then they presumably have to pick one or the other.
You can also argue that on-device seems creepier or more invasive, even if it doesn't result in Apple examining your photos, which is a reasonable reaction. It certainly breaks the illusion that it's "your" device.
But it's a fact that the on-device model, as described, results in less prying eyes on your iCloud photos than the on-server model.
[0] I'm not claiming this is the case, just saying for example
If we're looking at the end result, a mitigation of a loss of privacy is an increase in privacy compared to the alternative, no?
I mean clearly what you're saying here is "scanning always bad!". I understand that, I really do. I'm saying that scanning was never not on the table for a large corporation hosting photos on their server. Apple held out on the in-cloud scanning because they wanted a "better" scanning, and GP's point is that it's ironic that the one cloud provider willing to try to make a "less bad" scanning option is the one most demonized in the media.
None of this is to argue that scanning is anything less than a loss in security and privacy. Yes, yes, E2EE running on free and open source software that I personally can recompile from source would be the best option.
I guess you could say that in the same way that you can say that a gambler who just won $10 is "winning", even though their bank account is down $100,000. It only works if you completely lose all perspective.
I think that may be because it's far from clear that Apple's solution is "less bad".
Every single photo you upload is getting scanned -- it's just that Apple is doing the scanning on "your" device instead of their servers.
From the point of view of the privacy of your photos, I fail to see what the difference between the two is. I mean, if they did the exact same type of scanning on their servers instead of your device, the level of privacy would be identical.
In terms of general privacy risks, not to mention the concept that you own your devices, there is an enormous difference between the two, and on-device scanning is worse.
Good point. The question is privacy "from whom". For me, privacy "from apple" means mostly from malicious individuals working for Apple (indeed, if you have an iPhone running proprietary Apple software, you could never truly have privacy from Apple the corporation).
There are documented cases of employees at various cloud services[0] using their positions of power to spy on users of said services. Performing on-server scanning implies that servers regularly decrypt users' encrypted data as a routine course (or worse, never encrypt it in the first place), which provides additional attack vectors for such rogue employees.
On the other hand, taking the on-device scanning as described, the on-device scanning process couldn't be possibly used as an attack vector by a rogue employee, since Apple employees do not physically control your device. Maybe an attack vector here involves changing the code and pushing a malicious build, which is a monumentally difficult task (and already an attack vector today).
[0] https://www.telegraph.co.uk/news/2021/07/12/exclusive-extrac...
For me, privacy means that I have control over who I disclose what to. But context matters. If I'm in my own house, I (should) have almost total control over disclosure. When I'm in someone else's house, I have very little control as I'm subjecting myself to their rules.
A smartphone is probably the most intimate, personal device most people will ever own, and it's the equivalent of their house. However, if you're using cloud services, then you're in someone else's house and are subject to their rules.
That's why, in my view, doing the scanning on-device is not only dangerous, but unethical. Doing the scanning on the servers is neither of those things.
I get the argument about rogue employees, but I don't find it persuasive. I'm told that Apple keeps your data encrypted on their servers, although they hold the keys. If that's so, then "rogue employees" are something that Apple can, and should, control.
Arguing about where in the combined system the processing happens is shifting the deck-chairs on the Titanic, it's not making one part of the system more or less responsible than any other part.
CSAM scanning stops common proliferation of already identified materials and can help keep those from casually entering new markets. It does not protect children from being newly exploited by people using these services unless the services are also doing things they claim not to be doing.
In that case, Apple's claim of critical importance, if we interpret that as any more than rhetoric, doesn't mean what they imply it to mean.
Edit: I replied before you edited your comment. Leaving this one as is.
While I only know about this through anecdotes people share, I understand this systems generate leads so that agents (usually federal) can infiltrate groups and catch people uploading new material, thus preventing them for further victimizing minors.
Scaling up detection also makes it more tempting for bad actors to seed more content to fabricate the appearance of an epidemic. We already have inaccurate gunshot detection systems, field drug tests, and expert bite mark analysis being used to convict people. Would a jury even be able to examine the evidence if it involved CSAM?
The prosecutor will hire expert witnesses who will explain that their extensive training and education has led them to believe that the evidence is certainly illegal material.
Not sure if you are honestly asking about the jury thing but judges do have the legal superpower to see illegal images and they will usually instruct the jury to vote based on what they saw.
The US federal prison system being one of the biggest bullies on earth though, this cases rarely go to trial.
It’s not only a matter of Apple’s openness about it. It’s the fact of what they’re announcing.
The fact that cloud providers aren't building their systems in such a way that they are unable of conducting such spying is cause for concern. The fact that they are conducting such spying is outrageous.
One of the most common is “scanning” your location constantly, feeding your daily routines to marketing data aggregators, which is arguably more invasive of the average person’s privacy than CSAM flagging.
I’ve posted in the past a list of a short-list of offending SDKs frequently phoning home from across multiple apps from multiple developers. Since weather apps sending your location surprised people, I thought this problem would get more traction.
This Apple thing, where this is iCloud file upload client SDK doing a thing on upload, is an instance of this class of problem.
It’s not an Apple thing, it’s a client SDK thing, and the problem of trusting actions on “your” end of a protocol or cloud service is not a solved thing.
It isn't the same process, it's an on-device scanning process and Apple is the first to implement this. Had Apple said they were scanning iCloud directly nobody bat an eye (I, for one, assumed they already did).
I don't think this is a relevant distinction. If the device had known the result and just sent Y/N to Apple, what would change in your argument? Nothing. Your last sentence would be just as arguable.
>Isn't this like "scanning in iCloud" but without Apple needing to have a decrypted photos in their servers?
Note that Apple right now already has the decrypted photos since they have the decryption key. There's no evidence they are even considering E2EE right now, and since there are other legal requirements for scanning (e.g. terrorist material) I'm not sure this allows them to implement E2EE.
And I don't see how the client-side feature can remain as is. There are obvious cases that would be caught by the typical server-side scanning and won't be caught here, so once the door was opened, the government will pressure. For example:
* What happens when the NCMEC database updates? It could be that the phone rescans all your images. Or that apple keeps the hash and rescans that. Note that the second case is identical to some server-side scanning implementations.
* What happens when you upload something to iCloud, delete it from your phone and then the NCMEC database updates?
If Apple keeps the hash, it's just server-side again. If Apple doesn't keep it but uses client side scanning, the phone has to keep the hashes so you can't ever truely delete an image you've taken. If there's no rescan, the bad guys get to keep CSAM on iCloud so long as they passed the original scanning with the older NCMEC database - surely the government wouldn't accept that.
(I considered scanning on download but I don't know if Apple/the government would like this compromise, since with that approach CSAM can remain on iCloud undetected so long as it's not downloaded, anyway it's not in the original papers).
Basically, either they scan on client-side more than they let on or we end up in a world with both server-side and client-side scanning. The latter is arguably worst of both worlds, the former has implications which need to be looked at.
I would argue that 'scanning' is the process of collecting data, not necessarily of interpreting it.
Apple takes a photo, runs it through some on-device transformations to create an encrypted safety voucher, then it gets "interpreted" once it's uploaded to the cloud and Apple attempts to decrypt it using their secret key.
Google uploads a raw photo, which itself is essentially a meaningless value in the context of identifying CSAM, and Google "interprets" it on the server by hashing it and comparing it against some database.
In both cases, the values that are uploaded by the respective companies' devices don't mean anything, in the context of CSAM identification, until they are interpreted on the server.
Some of Apple’s management must ponder that if only they didn’t go to such lengths in denying themselves access to user data, they’d have it so much easier—given no other takers of such a challenge among mid- to high-end device manufacturers, we all would probably have settled for e2e just never being a feature of commonly available cloud services. I wonder how successful the privacy-focused promotion had been for Apple; they seemed pretty OK making people want their devices for all the other reasons.
This is wrong, photos (and most data on iCloud) is not e2e: https://support.apple.com/en-us/HT202303
Not really. I'm just saying it's laughable to think that CSAM scanning in its current form is critically important. Maybe it's critically important for feelgood and PR but not for actually preventing child abuse. It's almost like saying scanning for catalogued images of violence stops violence. If it only were that easy, damn, the world would be a good place.
Now as to what online providers should or shouldn't do, I can't say. But a part of me hopes that they continue the march towards egregious censorship and privacy violations so that people would eventually 𝐖𝐀𝐊𝐄 𝐓𝐇𝐄 𝐅𝐔𝐂𝐊 𝐔𝐏 and realize that there's no way to have freedom and privacy when you're handing your life to corporations and proprietary software. I really do wish for a world that is de-facto free software and fully controlled by the user.
As for iCould, I have no skin in the game. I've never used an Apple product.
Once the on-device scanning Pandora's box is open, it's trivial for governments to request more stuff to be added to the databases and Apple can't claim the defense of "we have no such ability currently" anymore.
If I was paranoid I'd wonder how much astroturfing is going on here.
This is a mischaracterization of many of the arguments.
short_sells: My goal here is meta-goal: I am not trying to change your mind on this issue; rather I want you to acknowledge the strongest rational forms of disagreement.
This is a complex issue. It is non obvious how to trade off goals of protecting kids, protecting privacy, minimizing surveillance, catching predators, and dealing with the effects of false positives.
It is too simplistic (and self defeating) to think that just because people disagree with a particular conclusion that they are not allies with many of your goals.
So what is it actually trying to accomplish?
I really struggle to believe that they are trying to protect kids (out of the goodness of their hearts).
The only explanation I can think of is that this is some attempt at appeasing government agencies.
> but we have seen in the past that privacy once lost is nigh impossible to regain
Yes, this is a key argument in the mix.
Some follow up questions:
1. As I understand it, here on HN, we are an international audience of various ages. With that in mind, I don't know your contextual experience. Are you scoping this (a) to the internet era (roughly 2000 to present)? / (b) to particular countries?
2. The statement quoted above is stated as if it is a fact, but I hope you realize it is actually a prediction. What is the historical context for this prediction? How far out into the future are you predicting?
3. Can you pin down your prediction more precisely? What does "nigh" mean? (There is a lot of variation in what "approximately" means to different people. Often the 'exceptions' are quite informative.)
4. The argument, as written, is quite general, which makes it hard to discuss. Whose privacy and in what context? Chinese citizens searching the internet? Journalists doing investigative reporting? Americans shopping in surveilled supermarkets? (Think of this as an opportunity to explain)
5. Do you mean all of the above? If you do, yes, people say that online privacy has eroded in many senses. At the same time, the tools for encryption have become more powerful, understood, and used. My point: if you make a very general statement, it is only fair if you cover the full range here.
In summary, with the above questions, I want to both better understand you -and- push back too. Unfortunately, I don't find the discussion chain above (the ~3 ancestors) particularly persuasive. I say this even though I agree with some aspects of it.
So you know where I'm coming from: in almost all situations, I've found it is more effective to understand, discuss, explain, persuade rather than 'writing off' a group of people because you don't really understand them.
P.S. I've addressed your other points in a sibling comment.
This form of question, as written is unnecessarily limiting ...
(a) there doesn't have to be one thing that Apple was trying to accomplish
(b) there doesn't have to be one motivation
... so I'm going to respond to the spirit of the question with a set of explanations, all of which may be true (to some degree) at the same time.
- parents are fearful of their kid's online activities;
- parents are open to trying new ways to give their kids freedom with some guardrails;
- yes, many people at Apple do want to protect kids out of the goodness of their hearts. This fundamental instinct is widely shared, particularly among parents.
- putting the 'why' aside, many customers perceive value and will pay for it;
- Apple executives are mostly profit-seeking (with certain constraints such as: mental models, brand constraints, regulations);
- shareholders seek profits and generally have less loyalty to any particular company's 'values' -- meaning they will 'shop around' for the best performing companies;
- as a group, shareholders see mostly upside and little downside -- don't perceive significant direct harm from these changes (at least, not until this became a public relations issue);
- generally, corporations benefit from playing nice with the U.S. government;
- Apple, in particular, has quite publicly pushed back on law enforcement's calls for decryption;
- in particular, with heightened scrutiny of the large tech companies, olive branches are particularly useful;
- some at Apple may prefer to lead with a proactive solution rather than wait for imposed regulations;
- some at Apple see this as a proactive branding opportunity;
- some Apple engineers are at the top of their field regarding encryption, security, etc and may have deemed their offering the best practical option available;
- some at Apple certainly understand the risks but assess the balance of false positives and false negatives differently than you do;
My hope is to make it a bit easier to recognize the complexity here. Though it may be true that organizations act as one entity, it is not true that they have singular intent. Attempts to claim a singular intent or goal are subjective interpretations.
Note: the list above is presented sequentially, but I am not claiming any causal ordering. They would be better presented as a network/graph connected by topics and relationships.
Ah, good old apophasis:
> a rhetorical device wherein the speaker or writer brings up a subject by either denying it, or denying that it should be brought up. - https://en.wikipedia.org/wiki/Apophasis
In relation to https://news.ycombinator.com/newsguidelines.html :
> Please don't post insinuations about astroturfing, shilling, brigading, foreign agents and the like. It degrades discussion and is usually mistaken.
If one was paranoid, they might wonder if you forgot to switch accounts /s
How so?
And that's what Apple has announced. Comparing that to:
> "No other company is scanning for CSAM on your phone"
Suggests that you think the Apple system is scanning more than what you upload to the cloud, which it isn't. Or, less charitably, suggests you /want people to think it is/. Your other comment here https://news.ycombinator.com/item?id=28405476 suggests the same. And your comment here https://news.ycombinator.com/item?id=28405502 the same again.
no, they're just doing it in the cloud, after everything was uploaded automatically :)
https://www.missingkids.org/content/dam/missingkids/gethelp/...
Hundreds of thousands of people die each year due to filthy criminals driving too fast. Introducing this measure will save lives.
If you are uncomfortable with this measure, you drive too fast.
Over ONE HUNDRED people die per year from people not coming to a complete stop at stop signs, and at Apple, we care!!!!!! We'll never let the screeching voices of the minority stop us from invading your privacy for no good reason. We like the screeching, it makes us feel important and relevant. As the inventors of a small computer with a battery in it, we consider ourselves better than God.
In other words, we could improve the driving experience and save lives with a little bit of work if we had folks with a working brain in government and elsewhere.
https://about.fb.com/news/2021/02/preventing-child-exploitat...
"We found that more than 90% of this content was the same as or visually similar to previously reported content. And copies of just six videos were responsible for more than half of the child exploitative content we reported in that time period."
"we evaluated 150 accounts that we reported to NCMEC for uploading child exploitative content in July and August of 2020 and January 2021, and we estimate that more than 75% of these people did not exhibit malicious intent (i.e. did not intend to harm a child). Instead, they appeared to share for other reasons, such as outrage or in poor humor"
So a lot of this is memetic spreading, not pedos and child abusers sharing their stash of porn. And people don't react to child porn by spreading it like a meme on Facebook, so what are these pictures that get shared a lot? What's happening is people find a funny/hilarious/outrageous picture and share that. Funny moments might happen when kids play with pets, for example.
The other part is consenting teens sexting their own photos. And then there's some teens (e.g. 17-year-olds, which by the way is old enough to consent in some countries) getting accidentally shared along with adult porn by people who don't know the real age.
https://research.fb.com/blog/2021/02/understanding-the-inten...
"Unintentional Offenders: This is a broad category of people who may not mean to cause harm to the child depicted in the CSAM share but are sharing out of humor, outrage, or ignorance.
Example: User shares a CSAM meme of a child’s genitals being bitten by an animal because they think it’s funny.
Minor Non-Exploitative Users: Children who are engaging in developmentally normative behaviour, that while technically illegal or against policy, is not inherently exploitative, but does contain risk.
Example: Two 16 year olds sending sexual imagery to each other. They know each other from school and are currently in a relationship.
Situational “Risky” Offenders: Individuals who habitually consume and share adult sexual content, and who come into contact with and share CSAM as part of this behaviour, potentially without awareness of the age of subjects in the imagery they have received or shared.
Example: A user received CSAM that depicts a 17 year old, they are unaware that the content is CSAM. They reshare it in a group where people are sharing adult sexual content."
So there's reason to think that the vast majority of this stuff isn't actually child porn in the sense that people think about it. It might be inappropriate to post, it might be technically illegal, but it's not what you think. And if it's not child porn, you can't make the case that it's creating a market for child abuse. By reporting it, you're not catching child rapists.
I don't have a reference handy but I recall reading about the actual child porn that inevitably does get shared on every platform, much of it is posted by bots over VPNs or tor. So its volume isn't representative of the amount of child abusers on the network, and reporting these accounts is not likely to lead to anything.
Also: In May 2019 the UK’s Independent Enquiry into Child Sexual Abuse heard that reports received by the National Crime Authority from the United States hotline NCMEC included large numbers of non-actionable images including cartoons, along with personally identifiable information of those responsible for uploading them. According to Swiss police, up to 90% of the reports received from NCMEC relate to innocent images. Source: https://www.article19.org/resources/inhope-members-reporting...
Out of those remaining 10%, how much leads to convictions? Very little. Sorry can't dig up a source right now. Point is: millions of reports lead to mostly nothing. Meanwhile, children continue to be abused for real, and the vast majority of those who want to keep producing, sharing, and storing such imagery surely have heard the news and will find a different way to do it.
Platform operators are incentivized to report everything, if they're playing the reporting game. Tick box "nudity or sexual conduct", tick box "contains minors"? Report it.
It doesn't help the discussion at all that everything involving nudity and minors (even cartoons) gets lumped together as "CSAM" with actual child porn produced by adults who physically abuse kids.
I don't think the NCMEC shares numbers about how many of their reports result in actions by law enforcement agencies. They probably also don't really know, its kind of like a dragnet.
Also under the current "I'll know it when I see it" CSAM doctrine in US courts cartoons can actually be illegal and cartoons depicting the abuse of children are usually banned on most big media sharing platforms in the US, and most companies in the US won't host you or let you serve ads if you have that material. So yeah, it's not only muslim totalitarians that are ok with banning cartoons and punishing people for drawing them or sharing them, Uncle Sam may also send you to jail and deprive you of your rights if you draw the wrong thing.
I'm not surprised. Quantifiable accountability can be extremely problematic if you aren't actually making meaningful impact on the problem that you are claiming to help solve.
Stupid child abusers who put known-pornographic images in their iclouds are still child abusers. The fact that fruit is low hanging isn't a reason not to pick it.
It's only production of new/novel CSAM that harms children. Sharing of existing, known CSAM (what this system detects) doesn't harm anyone.
1. There is not much overlap between people sharing pictures from the database and people sharing new child porn that is not in it.
2. People sharing child porn are a lot more smarter or security conscious than the average person.
I'm genuinely curious about your thoughts but to be clear my focus is on this very narrow, nigh nitpick tangent.
See war on drugs or piracy, or alcohol prohibition for instance.
Now there's a million ways to share pictures online in a manner that bypasses the few big platforms' interference, and you really don't need to be a genius to use (say) password-protected archives. That's how a lot of casual online piracy happens.
This thing does very little to prevent spreading CSAM pictures, and it does nothing to prevent actual child abuse.
Which brings me to your example of a password protected archive. I'm ignorant on specifics so I'm just going to sketch a broad argument and trust that you'll correct me if my premise is incorrect or my argument otherwise flawed. Essentially, if something is opt-in instead of opt-out, a non-trivial portion of the population won't do it. Especially if it is even slightly technical, there are just a lot of people who will just stop thinking as soon as they encounter a word they don't already know. So if the preventative measure is something that is not automatic and built in to whatever tool they are using to share images, then that preventative measure will not protect most of them. So to bring it full circle, I think it would do much to prevent the spreading of CSAM because other bans have been effective and I don't think most people have the technical literacy to even be aware of how to protect themselves form surveillance. As you say you don't have to be a genius, but I'd suggest you'd need to be above average, which gives you over half the population.
Also thanks for responding, I hope I'm not coming across as unpleasantly argumentative, I mean all this in the most truth-seeking sense of a discussion (I was going to say 'sporting sense of a debate' when I realized I had never been part of a formal debate team and that might not mean what I thought it meant, heh).
Are you talking about a state where cannabis was de-criminalized?
I agree that ease of access does have an effect on people, but the effect of iCloud scanning is so marginal in the grand scheme of things that it's almost like fighting drugs by installing surveillance cameras in malls. They just go trade and smoke elsewhere. The friction is virtually zero, but the privacy concern of scanning on half a billion Apple devices is much worse.
It's worth keeping in mind that CSAM is already highly illegal and banned, whether Apple scans iCloud photos makes no difference on that front. So it's nothing like the difference between weed being de-criminalized or not.
Also, fact is you already have to jump through hoops to obtain CSAM. It's very rare to stumble upon it being casually shared online (I think the last I witnessed it must've been around 15 years ago on 4chan, and somewhere between 5 and 10 years ago in a spam attack on freenode). Trying to search for it on the clearnet is mostly going to yield nothing.
In general, people also tend to know when they're doing something highly illegal. And yet they still do it, just taking the steps to try avoid being caught. No difference with CSAM. They will jump through hoops, and "don't store child porn on iCloud" is the tiniest of hoops to jump for real.
Password protected archives were meant to be just one example of how to bypass scanning on cloud platforms, and one that happens to be widely used among casual pirates. Google drive might be one of the biggest pirate services around these days. The bigger point I was trying to make is just that there are countless ways to share files without exposing their contents to scanning. Nothing for people who are willing to jump through hoops to get CSAM.
Finally, one point I've had to try make over and over again is that detecting the storage or distribution of old (catalogued) CSAM photos is only very tangentially related to actual abuse of childen. Unfortunately that keeps happening even if you destroy the internet and make sure no photo is ever stored in the cloud again.
I've said it before: child abuse and violence existed before cameras and internet, and will continue to exist. Detecting images of abuse or violence is not going to stop abuse or violence.
And if someone makes a system that is efficient at detecting all catalogued (=old) images of CSAM, then that might just create a larger market for "fresh" (uncatalogued) child abuse. Credit to nullc for realizing this.
And on protecting-the-children front, there are much bigger problems than stashes of old CSAM. Like grooming, or chatrooms where child prostitutes are forced to stream for an audience..