Can't believe this dude is running software at Apple.
The headline for all of this should be "We know what it does and we don't want it"
Everyone understands well enough how it works and understands how trivially it could be changed to search for other kinds of content.
At some point, the back story behind this will leak out. That will be interesting. This has to be something Apple was asked/pressured/ordered to do. As a business activity it makes no sense.
Bear in mind that over the last week, people in Afghanistan have been frantically trying to erase any evidence of doing things the Taliban doesn't like, such as women playing sports.
I firmly believe we'll never discover who applied that pressure, or how. And it's hard for me to imagine the US government doing that without some court battle with serious risk of leaks to journalists. I mean, it makes no financial sense, so Apple would fight that. The only thing I can imagine is pressure from a private interest whose market is big enough for Apple to care. And if it is that market, then I think we're in for a tough ride until we can build our own devices.
If you read the documentation on how it's implemented, there's some fairly advanced crypto - private set intersection, threshold secret sharing - that only makes sense if Apple took the line "we have to do this, but we're willing to do it in a really expensive way so that we ourselves have as little access as possible". They went to the effort of running the NeuralHash on the client device, as far as I understand.
The standard implementation on other cloud providers is that the provider has access to your data on the server if they really need it - much cheaper, and makes it technically possible in future to easily change the T&C to "we may use this for market research and to improve our products". I view Apple's client-side implementation as drawing a very big line in the sand saying we will not do that and we are willing to put our money where our mouth is by writing this crypto protocol.
But that certainly raises the question, why would Apple do anything like that at all, unless there's some (current or anticipated) external threat that forced them to?
As to whether we'll ever discover it - I wouldn't have predicted the Snowden revelations, maybe this will come out the next time something like that happens. If it does, I also predict we'll be shocked to discover just how big the market and distribution network for child abuse online is, and how many people are involved.
I don't think it is big compared to what authoritarian states would like to use this for. You're talking about, at the very least, putting everyone in non-democratic states under even more pressure to obey than they're already under. And in democratic states this invites bad actors both in and outside the government. There is no net-gain in justice here. It's wrong across the board.
Welcome back to HN after taking 5 months off, btw. You're a SSC fan?
Yes, I agree the state market is even bigger - China's potential market for surveillance alone is 1.3 billion and rising - but I've heard from colleagues in tech who seriously think child abuse is only a very small number of bad actors. My understanding is the "industry" is almost as out of control as the illegal drugs trade.
I am indeed a SSC/ACX fan (and subscriber).
And now I am down a rabbit hole. Probably for a while. But thanks for that. Seems like the world needs more thinkers like this person, and everyone, including myself could learn a lot from them.
Snowden paid a very high price not many people can afford, especially when they have a family. So I highly doubt we could know it within a reasonable timeframe.
Anything going through a FISA court (Foreign Intelligence Surveillance Court) could come with a gag order, preventing anyone involved to talk about it.
> "Though these capabilities are intended to protect children and to reduce the spread of child sexual abuse material, we are concerned that they will be used to censor protected speech, threaten the privacy and security of people around the world, and have disastrous consequences for many children,"
Many people and organizations fear that Apple's plans are paving the way for governments to scan for more than just child abuse images. (e.g. https://appleprivacyletter.com/)
In that light, FISA is relevant. My statement was to show that not all dealings of the US government are public, because not all courts and legal proceedings are available to the public. So "serious risk of leaks to journalists" won't likely happen in that case.
This is a fairly typical corporate failure mode.
Sometimes the best thing is to do nothing as they have found out because now they’re stuck in a position where the two exits are either piss off everyone by writing off their privacy stance or piss off everyone else for canning child porn scanning.
What a complete fuck up.
(One counterpoint to this is that PhotoDNA has been used for scanning cloud images using NCMEC-provided hashes for a decade now and this slippery slope doesn't seem to have happened yet.)
My understanding is that there is a threshold that needs to be met, and then an Apple employee with a very difficult job will need to confirm the image. So even if NCMEC get pressured, I don't see what the result can be.
Replace “FBI” and “Biden” with their equivalents in various other countries and yes, it’s absolutely plausible.
I think it's not really a problem of what is being looked for, but the mere fact that it will be done without a warrant on a personal device.
Stop feeding them and they will starve on their ignorance. I'd rather be poor in a better world than rich in a dystopia.
The problem is that we all end up being poor in a dystopia. The regulatory capture of industries is an ongoing thing. When 20 years ago you could start your own business in your garage, without substantial capital, today you are dependent on "Angel investors" and bank loans, which is an equivalent of having to be referred into a "club". If you can't find an investor, it will take you a lifetime to save money to start a business, if you are lucky. Take notice that all taxation is being shifted to workers to limit their chances of raising capital and creating competition.
Not sure if it’s true or not, but we shouldn’t assume companies always intend their public plans to succeed. Maybe they actually want a high profile retreat — it could certainly save them development time down the road by not implementing government misfeatures.
Still incompetent. This would have made sense under another administration. The only thing holding back the antitrust dogs is public opinion. An Apple antitrust case would be politically costly in a way a Facebook case wouldn't. This type of thing corrodes the core of the pro-Apple vocal minority. It's bad GR, if that was the plan.
Neither do they about privacy.
There are so many more talented software development leaders out there (maybe scratch all the ones working on adtech), but he's clearly a cultural troglodyte and this is evidence that he is incapable of actually leading the future engineers and software workers of the world.
If I saw him in a social setting I would try to ask him how he lives with himself, actively enabling the potential future dystopia and being incapable of defending the actual impetus of his actions, much less the reaction to it. Truly unforgivable, especially in an age of increased social fragmentation.
He seems like a guy who genuinely believe that what he's doing is a net positive for society and that society should just leave things in his hands.
There are plenty of people like that at google last I remember. I used to go to the google dev day and the was no greater difference between really talented technical people presenting interesting things and engineering leadership believe that everyone should just have a dumb terminal in their hand and that google should do any processing and how that would be so much better for society.
There are good reasons not to want this. Just not making everyone into a suspect would be a start.
But whoever wrote the letter is clueless.
In my opinion, this is the biggest concern, not the technology. Before, Apple could simply refuse by saying we don't have the capabilities. But now, that excuse is gone. Apple's promise to human review content and only report CSAM is the weakest link.
could, yes, but given PhotoDNA has been using similar hashes from NCMEC for a decade and doesn't appear to have had similar surveillance pressure imposed, what makes the Apple scanning different to warrant slippery slope arguments like this?
Not with the hashes provided by NCMEC, though, right?
The same system is already in use for things other than NCMEC by other providers.
What do they have now that they didn’t have before?
So you'd have to pressure NEMEC and another org under a different government to both add the non-CSAM hash, plus Apple would need to be pressured to verify a non-CSAM derivative image, plus you'd need other hash matches on-device to exceed the threshold before they could even do the review in the first place (they can't even tell if there was a match unless the threshold is exceeded).
I get why people are concerned, but between this thread and the other thread yesterday it's clear that pretty much everyone discussing this has no idea how it works.
1: https://www.apple.com/child-safety/pdf/Security_Threat_Model...
Apple could have quietly implemented CSAM scanning server-side, and left the door open to it being quietly exploited in who knows what way. But they didn’t: instead they put a whole bunch of infrastructure in place that all but guarantees that they’ll be immediately caught (and publicly excoriated) if they try to use this CSAM mechanism for anything other than CSAM. (See the PDF that GP linked for technical details on why.)
Of course, they could still do it with some other mechanism. But in that case none of these CSAM changes are at all relevant to the concern, as that risk is unchanged from a month ago.
Aren't iCloud Photos already scanned for CSAM though?
Do you have some source to indicate definitively that they have not been scanning iCloud Photos for CSAM?
[1] https://web.archive.org/web/20190701000647/https://www.apple...
As it happens, I still think you're missing the point that myself and others are trying to make. Perhaps the following is a better way of phrasing my inquiry:
What guarantees to we have that Apple will, always and forever, only use the NCMEC/CSAM/IWF system to scan images on phones? By that I mean, isn't it perfectly plausible that Apple can leave the NCMEC/CSAM/IWF system in place, as it is described in your post and the linked PDF, and at the same time partner with [random body X/Y/Z], who has their own database of [whatever] that they scan separately from the NCMEC/CSAM/IWF system? These two different scans wouldn't ever need to communicate with each other, and could run at the same time.
So what? What you're describing is not what this system does. You could argue against literally any possibility anyone could dream up, because they're the platform vendor and they can make any change they want. If that's a problem when you're evaluating your threat model then this new system is the least of your worries.
And as far as I can see this system is the least intrusive version of what any of the other cloud vendors are doing. Don't want it? Turn off iCloud Photos and sync to something else like your own Nextcloud instance.
I said that the hypothetical system could be set up, and function, in the same way as the NCMEC/CSAM/IWF system, just with different entities behind it. Despite chastising me for what you thought was a failure to comprehend your post, this is the third time you seem to have been unable to grasp that concept.
That's part of what people are concerned about - an identical system running in tandem to this, but with less savory characters behind it. Given how many comments on HN have addressed that this week, including all of mine right here, I am at a loss for how to make that any clearer to you.
Right!
>Yes, a lot of comments have addressed something that isn't happening.
No, and the confusion lies in the tense, "isn't" vs. "won't". A lot of comments have addressed a lack of trust that such a scenario won't happen.
Yes they could do that, but this doesn’t help them.
Right! I clearly stated multiple times that this would be a second system working in tandem, of course it would need to be deployed on its own. I am entirely unsure how you think I am ignoring that when it's the premise of my argument and of the concern of others.
Edit: Removing some paragraphs, because if that's not perfectly clear to you by reading this entire comment thread, even when I basically said as much in the post you just responded to, then I really don't know what to tell you.
You being able to imagine a fictional evil parallel system isn’t a valid criticism of what Apple is actually doing.
Right! And where did I criticize them? All I did was ask, “What guarantee do we have that Apple will, always and forever, only scan for data that comes from NCMEC/CSAM/IWF?” and then gave an example of what such a scenario might look like when another user responded to me.
I expressed a concern (I even used that word, and not "critique" in my last post that you responded to) that this type of scanning could be expanded in the future, but I have not criticized them broadly nor have I criticized this specific program. How has that not been clear?
If you think I’ve done otherwise, I’d appreciate it if you could quote what you interpret as a critique, so that I may work to avoid such confusion in future conversations similar to this.
Otherwise why post it here? I doubt you’d claim that your ‘concern’ is irrelevant to this topic.
It’s an innuendo intended to imply a likelyhood of future wrongdoing.
It works like this: ‘What guarantee do we have that he won’t hit his wife after we leave?’
Obviously this is a way of suggesting that the man will hit his wife, phrased as a question to facilitate the same kind of denial that you are using here.
… duh? I never said it wasn’t related, I said that it’s not a criticism of the current program.
>Otherwise why post it here? I doubt you’d claim that your ‘concern’ is irrelevant to this topic.
Correct!
>It’s an innuendo intended to imply a likelyhood of future wrongdoing.
No, it’s intended to state clearly - not imply - a concern around whether or not Apple will be pressured by other bodies to begin scanning on-device content for hashes pertaining to subjects beyond child abuse.
The question is simple - what guarantee do we have that Apple won’t expand what kind of content they scan for? What is so difficult to grasp about that, and why do you, it seems, feel that that is an entirely irrelevant question that isn’t worth discussing, answering or being concerned about?
>It works like this: ‘What guarantee do we have that he won’t hit his wife after we leave?’
>Obviously this is a way of suggesting that the man will hit his wife, phrased as a question to facilitate the same kind of denial that you are using here.
You leave and hope that he doesn’t, but would you never follow up with your friends to see how they’re doing after a domestic altercation, to make sure it hasn’t happened again and that they’re safe from harm? What is so wrong about being aware of possibilities and remaining vigilant?
You’re both proving the point.
Nobody said the man had done anything wrong, and yet you are assuming he did simply because someone asked a question and recommending that people act on this false impression.
That is exactly the goal of innuendo.
But again, since you continue to avoid what I'm asking:
>The question is simple - what guarantee do we have that Apple won’t expand what kind of content they scan for? What is so difficult to grasp about that, and why do you, it seems, feel that that is an entirely irrelevant question that isn’t worth discussing, answering or being concerned about?
and perhaps more curiously
>What is so wrong about being aware of possibilities and remaining vigilant?
My scenario never said anything about ‘leaving’. You continue to confirm the point. You read that in to the scenario.
That’s the point. You completely failed to understand the scenario as written and made up your own story to suit your prejudices. The only violence was in your imagination.
What guarantee do we have that you aren’t doing the same thing with Apple?
LOL, surely you jest:
>It works like this: ‘What guarantee do we have that he won’t hit his wife after we leave?’
At this point, after continuously misinterpreting my posts (almost intentionally, as though you're arguing in bad faith), you've effectively moved on to gaslighting, so I'm just going to continue to reiterate the same questions I have been posting, which you continue to avoid answering. Carry on as you have been if you want, but this is about all you'll get from me moving forward until you answer them directly.
- Apple has decided to scan for a specific kind of content on their phones, what guarantee do we have that they won't scan for other content in the future?
- For someone who commented, "stand up for civil liberties now", why are you so opposed to people being aware of possibilities and remaining vigilant?
To be fair, if the other organisation is IWF[1] under the UK government, I don't think there'd be much pressure needed to get them to comply - just offer to bung them and their mates a few million in contracts and you'd be golden.
It's a sensible plan, it just might not be as strong as it seems.
And that interview is likely supposed to telegraph that they “don’t care”. It wasn’t a live one, was it? They could have improved parts of it if they wanted.
Anybody a link?
Rather than fighting small battles to defend privacy, just take a hyperbolic extreme and use the backlash to bolster their actual position.
(or I geuninely think they have lost their f'in minds)
Team Apple,
This is from someone outside the Apple ecosystem that management brought in and gave a lot of credence to.
I wanted to share a note of encouragement to say that everyone at NCMEC is SO PROUD of each of you and the incredible decisions you have made in the name of prioritizing child protection.
The recipients of this memo didn't make any of the decisions and those were foisted on them by management who in turn were heavily influenced by this person/organization. As to why management was so heavily influenced, I have no idea. This is an attempt to get buy-in from Apple employees I suspect, or at least to ease their concerns. "Don't worry, you are doing the right thing."
It’s been invigorating for our entire team to see (and play a small role in) what you unveiled today.
The team they are referring to are the ones that pushed for the implementation for child protection. The recipients did all the work because management told them to.
I know it’s been a long day and that many of you probably haven’t slept in 24 hours. We know that the days to come will be filled with the screeching voices of the minority.
This is the big quote. The author knew what this would do, and knows the tarnish the Apple brand is about to go through, but is trying to convince the audience that it's only a small group of people who will reject this and it will pass over. For the author, the ends justify any means because they are probably personally emotionally traumatized by child porn (rightfully so). Once this happens though, any rationality about consequence of action goes out the window. This is an attempt to get everyone concerned on the same level of ends-justify-the-means with the author. I don't know if the author believes this is just a temporary outrage or not. I suspect not. It doesn't really matter though the author got what they wanted, Apple brand be damned.
Our voices will be louder.
This obviously isn't the case. Not sure if the author means their organization or their organization + Apple. Either way, the counter resistance to this has been pretty minimal. They left Apple hang out to dry it seems.
Our commitment to lift up kids who have lived through the most unimaginable abuse and victimizations will be stronger.
During these long days and sleepless nights, I hope you take solace in knowing that because of you many thousands of sexually exploited victimized children will be rescued, and will get a chance at healing and the childhood they deserve.
This is the emotional sell, or "for the children." It's also a difficult bridge to reason across; comparing the possession of pictures and the manufacture of pictures as the same thing. Obviously the hash compare only finds existing manufactured pictures, so I don't see how it will protect children from any abuse that hasn't already occurred. Not only is this an emotional sell, it's a lie.
Thank you for finding a path forward for child protection while preserving privacy.
The final emotional pitch directed at people at Apple typically concerned with privacy. They aren't really protecting children and they also aren't preserving privacy. It's a boondoggle.
In closing, "for the children," is almost always a trap. It was used when I was a kid to mass incarcerate Americans that had drug problems, many of them the children they were protecting. It's typically done in an any-means-necessary heavy handed manner, as seen here. I don't condone child pornography in any way. I have never seen child pornography and hope to never see it. At the same time, what they are doing is the road to hell paved with good intentions seemingly by people who are so traumatized (based on their work) that they can't see the collateral damage they are doing.