I've read this whole situation as a signal to China and other authoritarian regimes that Apple has finally seen diminishing returns from the "Apple is more secure" angle and is now looking elsewhere for growth. It's just business.
I think what we're seeing is Apple betting on using cryptography as part of the product design phase. Apple devices already do weird things like wake up to announce their physical location so that users can find their devices. The thought of a powered down or suspended laptop waking up to announce its location isn't something I particularly want, but Apple users seem to like it.
Anyone who has spent any time on spaces that are strongly encrypted and focused on privacy know how quickly they become havens for the sort of material that Apple doesn't want associated with its brand. How many "Apple protects child predator" news stories do you think Apple can withstand while still remaining a luxury brand?
Apples goal here is to have the reputation for end-to-end encryption and privacy while simultaneously not being seen as a phone for child predators. They don't have a lot of options if they want to thread that needle.
I've thought about this space quite a bit, and all options suck. Client side scanning is really the only choice with reasonable tradeoffs. The other option is scanning encrypted photos on cloud using secure enclaves to do the scanning. My guess is that when the tech makes that possible Apple will move in that direction.
I agree that this isn't the best for privacy nuts like me. But the iPhone isn't a blackphone, it's a luxury handbag. The phone isn't for privacy nerds, the privacy is there to make other mobile OS's look cheap and tacky.
They deserve to be raked over the coals for this, there's no world where their current design is a "good" or "right" one.
Child abuse is a serious problem, but building a surveillance panopticon is not an acceptable solution to it. Better investment in education, health care, and reporting hotlines are the way forward to stop this issue at its source.
Five years ago, the idea of Apple scanning photos on your phone would have been absurd.
Five years from now, what will people think about hotels installing AI-powered cameras in every room? The vendor swears they only start recording when they detect an act of abuse. It sounds absurd now, but where do you draw the line?
It does not really matter if the scanning happens on device or iCloud in this situation, because you have to always trust their closed source system. Google has scanned your images since 2009 in the cloud unencrypted, but now when Apple makes situation better, it is suddenly bad. All tools have been out there already. There are no really other options to get more privacy than this, but people refuse to see that.
Well, there is voting. Vote people who puts privacy over everything. That would make everything easy.
At the moment Apple's scanning policy is about the same as it was before. They claim they're only scanning photos if iCloud photos are enabled. The change they're advertising is doing the actual scanning process locally.
The problem is two fold. The first is Apple went from scanning only explicitly uploaded content to local content. Since they've decided to intrude on local content once "for the children" it's not out of the realm of possibility (if not likely) they will make further intrusions in the future for prima facie noble reasons. Are third party apps going to be restricted on saving data unless they allow access to Apple's CSAM scanner? Will it start scanning texts or e-mails tomorrow letting and rando flood a person's phone with CSAM and get them arrested? Adding a local scanning system like this is a slippery slope.
The second problem is the opaqueness of the system. This has multiple sub-problems. While the NCMEC has a laudable goal, involving them in the CSAM scanning process involves an outsize level of trust I don't think they have earned. They have law enforcement's unfortunate disdain for personal privacy coupled with a fanatical devotion to their cause. They believe their actions are always correct and just so long as they supposedly serve their goal of "protecting children".
Due to the opaque nature of their content library it's not crazy to think repressive regimes will get self-serving content added to the source libraries for CSAM scanning. There's plenty of places where homosexuality is punishable by death and even mildly anti-government content will land you in jail. Obviously you and I can't go look at NCMEC/ICMEC CSAM libraries to check for falsely added content. So how are we supposed to trust a system run by fanatics to not have simple errors?
Which leads to the other opaqueness sub-problem. Apple's design is interesting, if not laudable, but is closed source and full of black boxes. PhotoDNA, NeuralHash, and the like are not published algorithms anyone can verify. We don't even have a way of knowing if some image we have tripped a false positive and have to trust Apple's unknown "threshold" isn't 1. So not only does the public, the subject of these new intrusions, have no way of auditing the database but they have no way of auditing the code or process. A stupid bug in the scanning system could get a user reported to Apple which we then have to trust not to forward (and not to have additional bugs in their reporting system) them to law enforcement and ruin their life.
So I am concerned with scope creep and bugs/false positives. I can live with a bug that causes video playback to stutter or a black box system in Maps that gives me the wrong hours for a restaurant. It's much harder to live with bugs that can get me arrested or even killed thanks to trigger happy police. Apple's system might be technically adept but their promises of future behavior aren't trustworthy since they've already changed their behavior with this new system.
Major difference is, that they have no access for other images anymore as they used to have. They leave device as encrypted. Images used to be plaintext in the eyes of Apple.
> The problem is two fold. The first is Apple went from scanning only explicitly uploaded content to local content. Since they've decided to intrude on local content once "for the children" it's not out of the realm of possibility (if not likely) they will make further intrusions in the future for prima facie noble reasons. Are third party apps going to be restricted on saving data unless they allow access to Apple's CSAM scanner? Will it start scanning texts or e-mails tomorrow letting and rando flood a person's phone with CSAM and get them arrested? Adding a local scanning system like this is a slippery slope.
Emails have been scanned for long time in the cloud already. The rest is only speculation and against what they have told. It might be hard to trust, but in closed systems it is all we have. We should be worried when they actually say or start doing that.
> There's plenty of places where homosexuality is punishable by death and even mildly anti-government content will land you in jail.
It is fair to not trust third parties (NCMEC/ICMEC), but Apple is responsible for making the algorithm and testing that. Misuse must be part of their tests at this level. iCloud photos used to be plaintext so this hasn't changed from that perspective. If there is evidence that they are scanning other images outside of iCloud as well, then we should get the pitchforks and torches.
> We don't even have a way of knowing if some image we have tripped a false positive and have to trust Apple's unknown "threshold" isn't 1. So not only does the public, the subject of these new intrusions, have no way of auditing the database but they have no way of auditing the code or process.
This isn't true, since all math of their system is public and available on here: https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu... But code is as closed as always been. You have same level of trust for iMessage E2EE or even the screen lock of your phone.
Due to the way how system is expected to behave (it only looks existing matches from the provided data, with certain modifications), it is certainly possible that 1/1 trillion false positives is reachable, because they can validate it during development. They are not developing some AI to match totally new wild images. There is human validation, so nothing is automatically triggering police.
This is different from what your comment implies in two ways. First, they do not have an obligation to actively look for CSAM; they only incur an obligation if they find it. Second, the obligation applies to apparent illegal content rather than known illegal content. What qualifies as apparent could end up in court.
https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim...
This isn't that simple. If NCMEC comes with the properties of CSAM (e.g. hashes) and asks provider especially those to be removed from their cloud, it is hard to remove them without looking for them. This is different than an obligation to actively look for CSAM in general.
If NCMEC told a provider that a specific URL (or similarly unique identifier) contains CSAM, the provider would be obligated to destroy the associated file or be guilty of possession/distribution because at that point they know what they have. That's different from NCMEC providing hashes that could identify files the provider may or may not be storing.
Can you describe the two options they had?
Sorry, this was three options.
No form of apologetic or "technical" explanation can remove this from reality now. They are betting heavily on their "core" demographics to trust them automatically and without any form of critical thinking.
If this implementation has no effect on Apples bottom line. Things are over. We will live in badly implemented version of the Minority Report.
Not doing anything anti-consumer that the law doesn't force you to do is "radical"? I know you're not an astroturfer, but I had to double check because this is textbook astroturfing tactics.
Apple simply does not have to do this, as far as I'm concerned it's obvious they're either currying political favors or being incompetent. It's perfectly fine if they want to run it on their own unencrypted devices, they absolutely don't have to overstep into their user's devices.
* Think about what happens to CSAM uploaded to iCloud before NCMEC tags it. This has to happen for each new CSAM, since NCMEC can't tag what it doesn't see yet.
Surely Apple and NCMEC want to be able to catch these perps (which they easily would have with server-side). Doing it client-side requires expansion of scanning to do much more.
- Apple has over a billion devices out there.
- Child abuse is a rare problem, but with over a billion devices, there will be enough of it for a lot of newsworthy stories.
- Child pornography takes just one abused child for an arbitrary number of viewers. Arguably, by the time you're limiting the number of viewers, most of the harm has been done.
On the whole, I'm not quite sure how the Apple plan will protect actual children from rape (except to somewhat reduce the secondary harm of distribution). I can clearly see how it will protect Apple from bad press, though -- people won't use iPhones to record that.
On the other hand, an investment in education, health care, reporting, and enforcement could significantly reduce the amount of child abuse, but with 7 billion people in the world, no expense would bring it to zero. So long as it's not zero, the potential for bad press is there. Indeed, usually if something happens a few times per year, it receives more bad press than if it happens a few times per day.
Apple has every incentive to be (1) seen as doing something (2) do things which protect its brand value. Apple has no incentive to invest in education, health care, reporting, and enforcement. Those seem like good things to do, but if anything, if a scandal comes up, those sorts of things are used to say "See, Apple new, and was trying to buy an out."
As a footnote, if we value all children equally, a lot of this is super-cheap. This is a good movie:
https://en.wikipedia.org/wiki/Born_into_Brothels
And the problem it portrays could probably be solved with the same finances as the salaries of a few Apple engineers, and a focused, targeted effort to identify child prostitutes, help their families with the economics which force those kids to become child prostitutes, and get those kids into schools instead.
I'm guessing the $100k raised from this film will do more to protect kids than this whole Apple initiative will do.
You bring up the distinction between "possession offenses" (i.e., a person who has CSAM content) and "hands-on offenses" (i.e., a person who abuses children and possibly, but not necessarily, produces CSAM). Detecting possession offenses (as Apple's sytem does) has the second-order effect of finding hands-on offenders because hands-on offenders tend to also collect CSAM and form large libraries of it. So finding a CSAM collection is the best way to find a hands-on offender and stop their abuse. Ideally, victims would always disclose their abuse so that the traditional investigatory process could handle it -- but child sexual abuse is special in that offenders are skilled in manipulating children and families in order to avoid detection.
I think that the case of USA v. Rosenchein [0] is a good example because it shows the ins and outs of how the company->NCMEC->law enforcement system tends to work and how it leads to hands-on offenders. It's higher profile than most, perhaps because the defendant (a surgeon), seems to have plenty of resources for fighting the conviction on constitutional grounds (as opposed to actually claiming innocence). But the mechanism leading to the prosecution is by no means exceptional.
Caveat: Not a lawyer.
[0] https://www.anylaw.com/case/usa-v-rosenchein/d-new-mexico/11...
We are citizens of our country and we deserve a dignified existence. We are supposed to have rights, and they're being worn away, formally and informally, by our governments and megacorps acting like NGOs.
I'm sympathetic to the overwhelming horrors of drunks, drunk driving, violent actors, child abuse, child porn, economic crimes, etc.
I've done my calculus, and I got my vaccine and I wear my mask in the current circumstances of our pandemic. But in a similar calculus, what Apple has planned to subject a huge portion of our population to, by din of their marketshare in mobile and messaging. I personally can't accept the forces at play in this Apple decision, and I'm continually baffled by those who think this is overblown.
So what's next? There might be some time left to secure our rights on Mars...
No government, no police, no Wild West "run them out of town" option. You think they're going to want to spend $500,000 return flight cost to send potential criminals away or just "let them be" in an environment like that?
The idea that you might be able to go there and "demand your freedom" without being a billionaire owner of the colony is ill-thought-out. Subjects will have no leverage and no options, and leaders will have billions sunk into it and demand obedience like a Navy Submarine.
However, I'd rather voluntarily subject myself to a dictatorship like that than believe all my life I have rights that are sacred, only to look up and find myself in an authoritarian panopticon.
I do harbor fantasies of some day collaborating on a new system of government, or at least laying the groundwork. It's not going to be Musk's planet forever, and the first generation of Martians will be volunteers who want the project to succeed. Which makes it more like the 13 original colonies than the Wild West.
It's not all hard to find such places. Many children are abused at scale, globally. I think few of those kids are getting filmed or turned in CSAM.
I'm also not at all sold on your claim that hands-on offenders tend to collect CSAM materials either, but we have no way to know.
I am sold on the best way of reducing actual abuse involves some combination of measures such as:
1) Fighting poverty; a huge amount of exploitation is for simple economic reasons; people need to eat
2) Providing social supports, where kids know what's not okay, and have trusted individuals they can report it to
3) Effective enforcement everywhere (not just rich countries)
4) Places for such kids to escape to, which are safe and decent. Kids won't report if the alternative is worse
... and so on. In other words, building out a basic social net for everyone.
We would not accept having breathalyzers in every car.
Or to bring it closer to the child abuse problem: Would we accept cameras that take pictures of the occupants of the car to make sure that the minors in the care are not being trafficked?
lol, that's not up to us. It's in the infrastructure bill.
https://www.mediaite.com/news/infrastructure-bill-could-requ...
"If you want a vision of the future, imagine a boot stamping on a human face - forever." - George Orwell
How long until general computing is given up due to hackers and piracy ala The right to read(https://www.gnu.org/philosophy/right-to-read.html)?
Though requiring the driver to blow into a straw doesn't seem particularly "passive"--whatever that means.
But the text makes it seem like they would position cameras toward your face and do analysis on impairment indicators like eye movement.
Imagine a medical condition that makes it look like you are impaired. Now, you have to go to the dealer with a doctor's note to get this system disabled. Or when you want to rent a car.
Or, if there is a case when driving impaired would be better then the alternative. You and a friend are camping in the woods out of cell range, you both have some beers then one of you trips and gets a deep cut on the leg. Now you have to wait a couple hours before he can drive you to where you can get cell signal, hope you don't bleed out.
So the equivalent is that for every single trip you take, you must prove you are not under the influence.
If you're going to pay to use a hired car, expect to have to show the car hire company sufficient proof that you won't expose them to unnecessary risks. If you're going to pay to use a hired server to store your photos, why shouldn't you demonstrate to the owner that you aren't going to misuse their services or break their terms of service or break the law?
If you want to drive your car on your land, it doesn't need any of that.
So ... this is your hellish dystopia, your "boot stomping on a human face forever", Hertz rent-a-car?
[1] analogous to you using Apple's iCloud servers.
The public roadway, something I don't have a right to, is what I am accessing, just like the iCloud service.
https://www.drive.com.au/news/2008-nissan-gt-r-uses-gps-to-d...
Trucks have tachometers which track drivers aren't driving too long, and are taking sufficient breaks.
> "It will then connect to a DMV database that verifies the information is correct and then to the insurance database to verify coverage."
Wouldn't it be nice to know that if you're in an accident, the other party can't simply say "I'm not insured lol" and drive away and leave you and your insurance to pick up all the costs?
Funny you would bring that up. I think the new infrastructure bill requires that for cars built after 2029 (or some other "future, but not that far" date)
This is the thing that privacy advocates seem to ignore. Measures taken to reduce child abuse won’t reduce the circulation of whatever CASM does get created.
Some even seem to think, a la the ACLU, that viewing child abuse material is a victimless crime, and only the creators of the CASM should be punished.
Just to be clear, I am neither defending nor attacking Apple. I don't even own any Apple devices.
I'm just giving my interpretation of the dynamics behind what is going on.
I'd add that they probably consider the scheme to be better than the alternative (which is how others do it, including Google IIRC), namely checking photos once they have been uploaded. They have gone to some lengths to do more on the device instead of uploading user data, in Siri for example, but also Photos.app face recognition etc.
Privacy isn't a toy for nerds, though. It's not even a luxury item. It's a need and a right of all people. There is a good option: keep people's stuff private. It's the only option.
It's a while since the ruckus about privacy from techie types has penetrated the public discourse, and I think this is a very good thing. The non-tech-savvy people, if anything, overestimate the degree to which their privacy is compromised, convinced that every sound they make within earshot of their phone is scraped for ad targeting.
But not one of the people in my anecdotal dataset change their behavior on this basis, nor even seem to be particularly bothered by it. I don't think you can even chalk this up to technical ignorance. Bush's warrantless wiretapping had something like 40% approval, and that wasn't even transparent or consensual!
It really does appear there are a massive amount of people out there who look at the current cost/benefit tradeoff of compromising their privacy and decide that it's worth it. Awareness is still important, but I don't agree with your suggestion that everyone be effectively coerced into accepting the tradeoffs that you or I accept.
I don't agree with this characterization - it's too willful. To me, it seems more like a helpless coping mechanism. Since they "overestimate the degree to which their privacy is compromised", they resign themselves to not being able to do anything to protect their own privacy. The phone is listening to them, the satellites are tracking them [0], websites are recording them - basically every electronic device they encounter is not under their control. Their privacy is already gone.
Then, they watch TV and see actors using surveillance systems to capture Really Bad People. Since they've already resigned themselves to the collection, the only thing they have left is to hope that said surveillance results in things that are good and just. And when you try to bring up real-world problems, they revert to coping mechanisms of how it doesn't bother them - because if it did, they're still ultimately powerless to change anything.
To cross this divide, I think we need to give people actionable packaged-up solutions they adopt to protect their privacy. Part of the difficulty is that most people use their phone as their primary communication medium, and the phone ecosystem is a privacy dumpster fire. I don't have a recommendation for increasing phone privacy besides LoS+microg and also stop using your phone so much - do most of your communicating from a real computer running Free software.
Incidentally this is why this Apple news is so terrible - they had seemed to plot a course for more user privacy. Even with Apple retaining control, it could have let people see there can be boundaries. But now they've basically thrown away user empowerment in favor of putting a government agent on every phone. And so we're right back to the understanding of "everything I do is surveilled".
[0] I'm obviously describing their perspective. I've tried to explain to people that GPS satellites do not themselves track you, but rather let your phone figure out where you are. And by them taking an interest in the software on their phone, they could prevent it tracking their location. But I generally hit a wall of cognitive dissonance where the "satellite tracking" was really just some talking point, rather than something they think they could prevent.
I don't doubt that some contingent of the market feels this way, but I'm positing the existence of a large section of the market that truly doesn't really care that much about privacy. There's a reason that privacy advocates spend so much time arguing against "if you're doing nothing wrong, privacy doesn't matter", and it's because so many see big institutions (tech cos, banks, gov't) as detached institutions that for the most part do the right thing. It's the same reason that most people don't have a coherent sense of government's monopoly on legitimate violence and coercion: instead of grappling with the nuances and trade-offs of this bargain, it's easier to just model them as "the good guys".
Naturally, I'm going off of my perception here, as there aren't well-defined statistics that would give us a more reliable sense of the attitudes towards privacy that affect (or don't affect) people's purchase decisions. But a high enough proportion of my non-tech-employee acquaintances are unbothered by privacy concerns that I have to at least acknowledge that they likely represent a non-trivial segment of the market.
> they're still ultimately powerless to change anything.
This doesn't comport with my experience with these people. One finds niche cases here and there where the trade-off for privacy/autonomy provides a pretty decent ROI. I've occasionally been asked about some of these decisions of mine. In those conversations, the people I'm talking about don't look at these trade-offs and decide that the effort isn't worth the privacy benefit: they hear that the benefit is privacy and immediately go "oh this isn't relevant to me".
As the guy sitting on the client-side, how about “No”?
This doesn't work because secure enclaves only move trust from the software developer to the hardware manufacturer, who has the code signing keys to update the firmware on the secure enclave. Which in this case would still be Apple, or someone equivalently [un]trustworty and subject to external coercion.
Seems like there is a way after all: https://daringfireball.net/linked/2021/08/09/apple-csam-faq
"Could governments force Apple to add non-CSAM images to the hash list?
Apple will refuse any such demands. Apple’s CSAM detection capability is built solely to detect known CSAM images stored in iCloud Photos that have been identified by experts at NCMEC and other child safety groups. We have faced demands to build and deploy government-mandated changes that degrade the privacy of users before, and have steadfastly refused those demands. We will continue to refuse them in the future. Let us be clear, this technology is limited to detecting CSAM stored in iCloud and we will not accede to any government’s request to expand it."
(Reminder: if you don't trust what they say, you can't trust that they haven't been doing this for years already).
Many other American companies have done business with totalitarian regimes over the years. Maybe there's too much money in that market for Apple to pass up. Given the growth of totalitarian strong men across the world it's probably a growth market these days.
Payment may not be overt. It could also come in the form of access to markets. The deal might be that Apple must demonstrate the ability to help a regime hunt down dissidents before it can sell domestically, or they could be offered a break from otherwise onerous import or sales taxes.