Having said that, there is an enormous amount of misinformation and fear-mongering about a pretty tame change. This seems like so much ado about very close to nothing.
a) They optionally scan messaged photos for nudity using a NN if the participants are children and in a family (the account grouping), and the group adult(s) have opted in, giving children warnings and information if they send or receive such material. A+++ thumbs up.
b) They scan photos that you've uploaded to iCloud (available at photos.icloud.com, unencrypted -- in the E2E sense, effectively "plaintext" from Apple's perspective -- etc) for known CP hashes. Bizarrely Apple decided to scan these on device as well, causing 99% of the outrage and confusion, yet every major cloud photo service in the world does such checks for the same reason, whether you have the photo set to private or not, and presumably Apple decided to do it on device simply as free distributed computing, taking advantage of hundreds of millions of high performance chips, but most importantly as a PR move demonstrating that "Apple Silicon helps with Child Safety", etc.
That's it. Various "this is a harbinger of doom and tomorrow they're going to..." arguments are unconvincing. This does absolutely nothing to break or subvert E2E encryption or on device privacy.
EDIT: The moderation of this comment has been fascinating, going to double digits, down to negatives, back up again, etc.
https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
So now they’re doing it on device too. This feels like it’s putting in place the foundation to scan all offline content.
Moving it on device will show 0 improvement to the original goal, while opening a door that quite frankly I never expected Apple to be the one to open (I would have bet on Microsoft).
The CSAM scan is only for photos that are to be uploaded to iCloud Photos. Turning off iCloud Photos will disable this.
My point is that to my knowledge, this is the first time that an on device "content check" is being done (even if it's just for photos that will end up in iCloud). This is the precedent (the on device check) that makes me and some others uneasy, as pointed out in the linked letter. The fact that it applies only to photos going to the cloud is an implementation detail of the demonstrated technology.
Legislators around the world now have a precedent and may (legitimately) want it extended to comply with their existing or upcoming laws. This is not a particularly far fetched scenario if you consider that Apple has already accommodated how they run their services locally (as they should, they have to comply with local laws around the world in order to be able to operate).
That's the crux of the issue most of the people quoted in the letter have, one can argue it's just a slippery slope argument, I personally think that one can be legitimately concerned of the precedent being set.
Keeping doing it on server, in my opinion, was a much better option for users (with the same compliance to local laws and effectiveness to the stated goal as far as we know, there's no improvement on that front, or none that couldn't have been brought to the existing server check), and ultimately also a safer option in the long run for Apple.
They've opened themselves, for little reason, to a large amount of trouble on an international scale and at this point rolling it back (to server checks) might not make a difference anyway.
I imagine there are some frantic meetings at Apple today. However the grossly misleading claims people have been making to fear-monger aren't helpful.
Leave my kids alone.
If they want to share photos of themselves naked, it's none of anyone's business except them and maybe me (maybe), certainly not a huge American corporation.
Neither me or my kids have iPhones, but as others have observed, I have no illusions that Google will follow suit. Our options are becoming pretty limited at this point.
And it's a useful, valuable option that many (I would wager the overwhelming majority) parents will enable.
Apple made a huge PR mistake announcing both of these systems together (the CP hashing system and the NN message warning system), because as seen throughout the comments it has led to lots of people conflating and mixing and matching elements of both into a fearsome frankensystems.
I agree they could have slowly walked people through the components one by one, but so much of what is in the comments is pure bad faith that I am not sure that it would have helped.
By “bad faith”, I mean strongly asserting as true claims that people know they haven’t checked, and which later turn out to be false.
In a cynical way this might actually be better PR for Apple. They know they can’t prevent people who dislike them from jumping to the most negative conclusions possible. By presenting these features together and letting the crazy interpretations abound, they make their opponents seem unhinged, and this obscures the real but less apocalyptic concerns.
I am certainly willing to agree I conflated them if you clarify.
Neural networks are from perfect, and invariably parents and child are going to have a laugh when it triggers on a picture of random things.
The other, completely separate system checks files stored in iCloud photos against a hash list of known, identified child abuse photos. This is already in place on all major cloud photo sites.
Apple clearly wanted to roll out the first one but likely felt they needed more oomph in the child safety push so they made a big deal about expanding the second system to include on-device image hash functionality. I would not be surprised at all if they back down from the latter (though 100% of that functionality will still happen on the servers).
You will be flagged, reported to the authorities under technical argumentation the Algorithm has a "one in a trillion" chance of failure. Account blocked, and you start your "guilty until proven innocent process" from there.
Due to scale at what Apple works its also clear to see, if you think about it, that the additional human process will be on a random basis. The volume would be too high for a human chain to validate each flagged account.
In any case at multiple occasions, it is clear that the current model of privacy with Apple is that there is no privacy. It is a tripartite between you, the other person you interact with, and Apple algorithms and human reviewers. Is there any difference between this, and having a permanent video stream from what is happening inside each division in your house, analyzed by a "one in a trillion neural net algorithm", and additionally reviewed by a human on a need to do basis ?
CSAM
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
"Using another technology called threshold secret sharing, the system ensures that the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images."
"The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. This is further mitigated by a manual review process wherein Apple reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC. If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated."
So in other words, there is no doubt for this one there will be intervention, of Apple employees when required. For training purposes, for system testing, for law enforcement purposes etc...
I guess these and other functionality was the reason the decided not to encrypt the icloud backups
"Apple dropped plan for encrypting backups after FBI complained" https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
Now concerning the second feature/system called: "Expanded Protections for Children"
"https://www.apple.com/child-safety/pdf/Expanded_Protections_..."
After reviewing what I believe to be every single document published so far by Apple on this, I found only one phrase and nothing more detailing how it works. Looking at the amount of detail published for CSAM, the lack info on this one already looks like a redflag to me. So I am not really sure how you can be so certain of the implementation details.
The phrase is only this: "Messages uses on-device machine learning to analyze image attachments and determine if a photo is sexually explicit. The feature is designed so that Apple does not get access to the messages."
Nothing more I could find. If you have additional details please let me know.
This is a feature that will be added in a future update there some conclusions from my part. If we just stay with the phrase "The feature is designed so that Apple does not get access to the messages." I have to note its missing the same level of detail that was published for CSAM.
I can only conclude that:
- They do not get access to the messages currently due to the current platform configuration. Note they did not say the images will stay on the phone currently or in the future. Just that uses local neural nets technology and feature is designed, so that Apple does not get access to the messages.
- They did not say they will not update the feature in the future for leveraging the icloud compute capabilities
- They did not say if there is any opt in or opt out for data for their training purposes
- They do not say if they can update locally the functionality at the request of law enforcement.
I agree with you that it looks like it stays locally by the description, but the phrase as written looks like weasel words.
They also mention one of the scenarios: Where the child will agree to send the photo to the parent before viewing, would that be device to device communication or via their icloud/CSAM system ? Unclear, at least for what I could gather so far.
> If they want to share photos of themselves naked, it's none of anyone's business
What about if adult sexual predators want to share sexually explicit photos with your kids?
My understanding is that this feature isn’t about stopping kids from messaging each other - it’s about giving parents a warning if strangers are trying to groom them.
Also:
“ The Messages feature is specifically only for children in a shared iCloud family account. If you’re an adult, nothing is changing with regard to any photos you send or receive through Messages. And if you’re a parent with children whom the feature could apply to, you’ll need to explicitly opt in to enable the feature. It will not turn on automatically when your devices are updated to iOS 15.”
So basically the furore is based on a misunderstanding.
That would get a lot of people nervous. Let alone anyone smart who thinks through the implications here of how far the line is being pushed on how public your phone is.
"And it does so while providing significant privacy benefits over existing techniques since Apple only learns about users’ photos if they have a collection of known CSAM in their iCloud Photos account."
How would that help catch them if they can simply flip the switch?
No on-device photo scanning unless iCloud Photos is enabled. Isn't funny when you get the most important aspect wrong?
> How would that help catch them if they can simply flip the switch?
They never claimed that this would help "catch them" if they are not using iCloud Photos.
Although they're assuring us that they don't make mistakes. The technical term for that is either going to be "blatant deception" or "delusion". Apple are impressive but they haven't developed a tech that can't make mistakes.
Though I don't blame you at all: Read through the various hysterical posts about this and there are a lot of extraordinary misrepresentations.
I don't care. There is no way on this good earth that law enforcement is going to let them get away with that. They're claiming that they will be scanning things that are obviously child porn and ignoring it. That isn't a long term stable thing to be doing - if they think scanning for anything is ok there is no logical reason to stop here. So they probably aren't going to stop, and they certainly aren't going to announce every step they take to increase the net.
And their 1:1,000,000,000,000 number is still delusional. The system is going to produce false positives. There are more sources of error here than the cryptographic hash algorithm.
I have never once in my life thought about activating an automatic back up to cloud feature on any phone I have ever owned, for a single second. So yes, it is hella different. This is for all the same reasons I backup personal data only to my NAS and use cloud accounts for generic shit like purchased music backups and nothing more.
I prefer losing all my photos if my phone is pickpocketed in between backups to having a public record of everything I ever photographed. Am I the 0.00000001% or something? I didn't even realize I was the odd man out, honestly.
Given that this only applies to photos that are stored to the cloud, it seems like it has total relevance given that for users literally nothing has changed. To argue that there is some fearsome new development requires one to extrapolate into "Well what if..." arguments.
I don't see a reason to do this, other than to either scan all images, or claim that iCloud is e2e in the future.
And of course they went with the protection of children argument, which is bullshit. Apple gets paid by its users and should have no other interests than to get paid as much money as possible, regardless of who pays them.
That’s exactly why they are doing this. Providing a safe haven for child sex predators is bad for their brand.
Do it in the cloud just like every other service does. None of this anger would have happened if they just kept it in the cloud, and I truly can not fathom how this made it this far. I would peg overwhelming odds that they abandon the on device idea as it makes no sense and has brought incredible ill will.
e2e means something specific and should be an absolute requirement in a post Snowdon age, not something that is optional or a compromise.
We have no idea about the hash collisions. When talking about whole world, 2^256 isn't a big enough space.... even if they're using 256 bits.
And how dare anybody criticize this - criticism is tantamount to being for child porn. (Then again, that's why it was chosen. We'll soon see other things 'forbidden'.)
Such a collision would generate a human validation.
This applies to cloud-shared files. It actually has applied to cloud shared photos for years. It applies to literally every major cloud photo service.
2. is it really reviewed by a human? I see how YT works, and automated failure at scale is the name of the game
3. apple has said that the on-device scanning only provides a binary yes/no on CP detection. How do you defend against a "yes" accusation? (when stored on someone else's server, the evidence is there)
There would also presumably be human review involved in the legal process, e.g. law enforcement getting a subpoena based on Apple notifying them, and then using gathered evidence for a warrant.
The system is based on known image hashes, not arbitrary ML detection.
As this system is used only for iCloud photo uploads, the evidence gathering should be similar to that done by LE with other cloud hosting providers for years
lol picky I know
Clearly you do not understand the full ramification of what is happening here.
I will repeat: It is a very tame change. Were you frantic and delirious when a neural network first identified a dog in your photos? Isn't that the slippery slope to it reporting you to the authorities for something bong shaped?
Speaking of which, every bit of fear mongering relies upon a slippery slope fallacy. What is clearly a PR move is somehow actually the machinations of a massive surveillance network. Why? Why would Apple do that?
Because it gets required to by the laws of countries responsible for most of their market? And because authoritarian regimes intentionally blur the lines between criminal and political surveillance over time, making it harder for companies to draw hard policy lines? Concern about this doesn't require any bond villains, it just requires well-intentioned pragmatists on one side and idealogical politicos on the other.
FWIW, I think you have a point about the doom-saying. Countries with good judicial protections around privacy already use it as a backstop against dirty tricks where folks use one type of surveillance to require another. But it makes sense to wonder how those barriers will erode over time, and to worry about places where they don't exist.
Apple’s changes would enable such screening, takedown, and reporting in its end-to-end messaging. The abuse cases are easy to imagine: governments that outlaw homosexuality might require the classifier to be trained to restrict apparent LGBTQ+ content, or an authoritarian regime might demand the classifier be able to spot popular satirical images or protest flyers.
[1] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...
Apple - creates messaging platform.
Slippery slope - governments can force Apple to send them all messages.
Apple - creates encrypted messaging platform.
Slippery slope - governments can force Apple to send them the keys and all messages
Apple - Adds camera to device (GPS, microphone, accelerometer, step detection, etc)
Slippery slope - governments can force them to record whenever the government demands and send a live stream to the government. Or send locations, or walking patterns, or conversations.
Apple - makes operating system
Slippery slope - Basically anything. There are so many ways I could go with this. Government can force them to make a messaging and photo system, add cameras to their devices, entice users to take and accumulate pictures, and do image hashing and report suspect photos.
That's the problem with slippery slope arguments. They become meaningless rhetoric.
Image perceptual hashing is literally a single person's work for two hours. If you really think the barrier between an oppressive government stomping on groups or not is whether Apple did a (poorly communicated) Child Safety PR move and implemented this trivial mechanism...the world is a lot more perilous and scary than you think.
Also the difference here compared to the scenarios you've mentioned above is that Apple has walked pretty far. All it would take is to make the verification happen on all local files irrespective of its being uploaded to iCloud or not. Then any government can provide their own hashes to apple to keep track of. Apple won't have any idea what those hashes actually mean
No company can defend you from your government.
"All it would take"...
That is the slippery slope. If a government is going to say "that's a nice looking hashing system you have there, now we need you to..." they could as easily -- more easily -- have said "that's a nice filesystem you have there, we need you to...".
Hashing files and comparing them against a list is literally a college grad afternoon project. There is absolutely nothing in Apple's announcement that empowers any government anywhere in any meaningful way at all. It is only fear-mongering (or simply raw factual errors as seen throughout this discussion) that makes it seem like it does.
Right. Exactly. Any country in the world can mandate that Apple do anything they want (any of the slippery slope mandates), and Apple can comply or withdraw from the market. If any country wanted to demand that Apple compare all files against a ban list, they could have done that in 2007, or any year since. There is zero technical barrier and it would be a trivial task.
The point is that this development moves the bar infinitesimally. I would argue not at all. Fearmongering that depends upon "Well what if..." didn't actually think it through very well.
I guess this is where I differ with you. No government was able to preassure apple to implement a complete client side verification till now but who knows how things will be now that they have a system in place that can be easily modified.
Anyways I hope your prediction turns out right. I live in a place where privacy laws don't really exist. The last thing I want is any system that can be easily exploited by authorities to crush dissent.
The legitimate concerns here are not a slippery slop, they are real and self evidence born from countless examples through out history of these types of survelence systems being abused. EFF points to a coulple, other artiles point to other examples
It takes a person an extremely dense person (or intellectually dishonest) to simply hand wave all of these concerns away as "well that is just a slippery slope so I can ignore you"
I see Apple implement a trivial (laughably trivial) system -- a tiny little pimple on a massive platform of hardware and software -- and I don't think "this is it! This is what puts the surveillance state over the top!", I think "Hey look, trivial, constrained system".
But if you believe that this was what was between a surveillance state or not...well you must be much more intellectually capable than I and I am honored to be in your presence.
It is the very definition of a slippery slope. A journey of a thousand miles begins with a single step, but usually you're just walking to the bathroom.
I don't know what I think about the move, but I know this: intentions don't matter. Capabilities matter. If the current intention is benign, that means little.
As an Android user I'd love to gloat, after all Apple have really had the upper edge on privacy so far, and their users have not been shy about telling us. Again and again.
However any pleasure would be as short lived as peeing your pants to keep warm in winter, because if Apple proceeds, Google is bound to follow.
A user monitoring system like this is just ripe for abuse of the most horrific kinds. It must die.
This is exactly what my problem is. They are using my CPU, battery time and network bandwidth to do this when they should be using their own resources. I know I can turn off iCloud but I am paying for that and we already have an agreement in place.
Honestly, it's the only concrete thing to complain about. Every other complaint is based on a what-if, slippery slope concern.
Of course, as per usual, nobody agrees with me here on HN... but that's fine with me because they simply don't reply which lets me know that they don't have any good arguments against my PoV.
Anyway I agree with you, and have been curious what the neural net hardware on the new iPhone will be used for. Turns out it’s for checking my photo library for child abuse, how futuristic!
I have yet to hear a good counter argument though. If I had heard one, I would have changed my mind. So, I don't think I'm set in my ways. (I actually change my mind often when presented with new evidence!)
I will always consider downvotes without argumentation to be a lazy way of disagreeing and the people who do that I think are set in their ways. Until I hear otherwise, I will continue to consider my argument as superior.
Does anyone really change their mind until they hear a counter to what they already think? I don't think so... I already argued with myself and came up with my opinion on this, so really - I need external arguments to change my mind just like everyone else.
I kind of doubt it because I've been censored for simply asking very straightforward questions with zero adjectives.
No, what I gather is that many people cannot change their minds when presented with good evidence. Here's one: They complain about "monocultures" as if they're always bad but when I point out that the Linux kernel created a monoculture and the world hasn't imploded, they have no come-back. So they do the only thing that they can do.
It's fine with me, I wear it as a badge of pride because I know I'm right whenever that happens.
This is this a first step to "Detective in your Pocket", cloaked intentionally, or not, by a well-meaning objective. An objective we can all support. If you wanted to put in a thin wedge on distributed surveillance, where better to start? As pointed out CSAM filtering/scanning is already done so that’s not the issue here. There’s a big debate to be had, and being had, on the upsides/downsides and benefits/dangers of AI and false positives. That’s a huge issue in itself; but that’s not the biggest concern with this move. If Apple pushes on with this it’s a clear signal that they wish to march us all to a new world order with Distributed Surveillance, as a Service. A march in step with the drumbeat of your increasingly authoritarian (or if you are lucky or more generous, safety conscious) government.
I have signed the letter to express my very strong personal concerns but also as a CEO of a company that takes CSAM seriously and seeks to provide solutions, in search, without surveillance.
"Why didn't you tell me this was going on?" "You're simply too stupid to handle the idea that your betters might occasionally be untrustworthy"
I remind you that people made that argument about Facebook and it didn’t.
Raising awareness is about letting people know so that they might take the necessary precautions if they consider themselves to be at risk of its abuse, and degrades their faith in the institutions that support it.
That could mean anything from watching what they put on their iPhone, to putting them down the track of using degoogled android variants on select phones, to ditching cell phones all together depending on their needs.
Knowing that your phone is watching, reporting, and potentially moderating the content in it is information which is valuable in and of itself. Even if it only has utility to a fraction of the population.
We find ourselves in the tightening noose of a latent authoritarian technocratic surveillance society. Few people have anything to fear from it, or the resources to escape it. But some do, and should be given every piece of information that might help them moderate or escape it.
As I said, there are no good solutions.
* don't store images on your phone
* use a rom like Graphene, don't install a cloud storage or Google Play Services
Those aren't great solutions, but they're options.
If your goal is to inform a small minority of expert users that they should protect themselves against corporate/government encroachment, by the looks of the comments here, I’d say you’ve already succeeded.
This forum is too niche to fulfill that purpose on its own.
The mass distribution of the information, and arguments against apple's behavior should be intended to incidentally target relevant niches beyond technically expert circles of which the communicator is unaware.
That the argument and resolution of these issues is irrelevant to most of those who will be exposed to it is immaterial.
Your replies to my threads are as they were written.
I’m not missing your point if you are changing your point after I have replied.
I haven't touched any of my comments since they first received a reply. I often touch my comments within a couple minutes of posting in order to correct typos or clarify my intentions, usually by appending a paragraph. But I don't do so without remark once they've become canonical in a thread.
Your replies to my comments are as they were written.
Wait, no, that's not the argument ...
There is a difference between selling your data to the highest bidder through a stalker capitalism ecosystem and giving governments carte blanche access.
I am not at all advocating for the latter, but if you are fighting that battle, CP is not the hill to die on.
They were already sold into the Apple ecosystem with the iPhone 12's, M1 Macs and iPad Pros. They are going to have a hard time moving to another platform.
Best part? Apple is a huge customer of Google Cloud for storage so you can now tell Google about all the files on your iCloud account. [0]
'With privacy in mind.' /s
[0] https://appleinsider.com/articles/21/06/29/apple-is-now-goog...