Expanded Protections for Children
apple.com
apple.com
"We're going to scan your photos, on your encrypted device, to look for badness. Right now, we're going to claim that's only for the really icky people that nobody is going to defend, but, hey, once the tech is in place, who's to say we can't scan for dank memes and stuff?"
I think I'm done with Apple. Sad, really. I was hoping that their bowing to China with iCloud wasn't a sign of what's to come, but apparently it was. They had done such nice stuff with privacy too.
Demote my phone to a house phone and go without, I suppose.
I'm carefully not alleging that this is illegal, because it's a third party doing it and it's not obvious that they are acting as the governments agent. Regardless of whether or not that solves the legal problem of unlawful searches, it does not solve the moral problem that we have a right to be free from unreasonable searches.
Replace "child porn" with "political posters" here. If you would have a problem with that search, I claim that you should have a problem with this search, because the there is no evidence that the person you are searching is committing a crime, and as a result the claim for this to be a morally valid search needs to not be about guilt (which would require probable cause first), but about it not being a prohibited search in the first place.
Let’s walk through this.
1. Criminal kidnaps child and abuses him. 2. Criminal produced video of said abuse and sells it on the web. 3. Criminal continues to sell it and it spreads. 4. The video is detected by authorities who promptly add it to database. 5. Video is cryptographically hashed and now anybody who stores this content in iCloud can be identified 6. A customer of the criminal is caught 7. Forensics leads authorities to criminal who produced the video 8. One less criminal to profit from kidnapping and abusing children
Everyone tries to make this approach as a slippery slope to facial. It doesn’t have to be that way if the right people are in the loop to blow the whistle.
Full whitepaper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
This entire endeavor hands the keys to an unaccountable police state.
You are describing the reverse cause of targeted search for the individual in which each step has probability that is much less than 100%. The technology discussed is a broad sweep: everything, everywhere, every time.
Oh, well, no need to worry so much, Apple has just added data collection ability which others had in one way or another.
Most reductions of privacy toward the police would act as an effective deterrent to that crime and other crimes.
Deterring crime is not enough to justify a reduction in privacy.
For the second one, maybe, but I seriously doubt icloud encryption is going to do that.
Either way, just talking about baselines and percents is a good improvement over just "this would decrease crime". Add in the downsides too and you have yourself a good platform for discussion!
if you really want to put a dent in abuse, mandate cameras in every home because that is where most abuse happenes. Anybody who opposes that clearly has something to hide...
Police can always pinky promise to never use it for anything except catching the baddies.
Yes. For anyone who wonders. This is sarcasm.
If one child abuser can sell his contents to thousands via nothing but WhatsApp, word of mouth, and a Bitcoin wallet, what will you do to fight that?
I’m all for freedom of speech and freedom from fishing expeditions and what not, but if you don’t deter these things they will grow.
WhatsApps only claim to be private is end-to-end encryption of messages. The moment a polkce officer opens the phone of a buyer they can identify the seller. Same if an undercover cop hears the word-of-mouth and buys.
This is just ordinary good police work that I whole-heartedly support.
Don't think of an algorithm as an independent actor, think of it as an awesome tool that makes you 100x more productive.
a) Executives at Alphabet, who are ultimately in charge of the companies decisions
b) The individual product managers, lawyers, engineers, and whoever else is involved in creating, maintaining, and running the system that bans people inappropriately.
The algorithm they are using is a tool that they are using to do this, using a tool does not absolve them of the responsibility in any way.
... and guns and cars don't kill people, murderers do.
But now a company is creating s new kind of "gun" that huge chunks of the community of security professionals are warning about because it is insanely powerful and dangerous.
We have a number of reasons to warn about it:
- China will undoubtedly demand to have Apple scan for their hashes too. The databases these hashes are collected from are not public for obvious reasons so this means they can slip whatever they want into it. Maybe Apple will check the images before they send them over but that is not necessarily the case as far as I read it, also it takes one rogue employee to correctly classify as not abuse material but make a note of the account and send it back after work.
- This is not your average sha256 hash. This is perpetual hashes. They are made to catch not only the exact document but all kinds of variations of it. I have not specialized in perpetual hashes but as far as I can understand it goes without saying that the more resistant this is to modifications, the easier it becomes to create innocent images that triggers it.
- even if we had a magic algorithm that resonated with everything good and only matched the images we wanted there still is potential for abuse. When I was younger I browsed through the cache folder of my machine and I remember there being a lot of images there that I can't remember having seen on any site I visited. Now it is said that this algorithm will only flag images about to be uploaded so obviously cache folders won't be scanned. But once this tool is in place why won't governments start applying pressure to Apple to scan everything? And what prevents someones soon-to-be ex from downloading and slipping some into iCloud when one leaves the phone unlocked?
- Due process should sort many of the problematic cases here, but child abuse is for good reasons one of the worst things you can get accused. A mere accusation is often enough to ruins someone's life even if it later becomes clear to law enforcement that the person is innovent. Also most places have a way to go wrt due process.
In the E.U., and other places, companies are definitely held to stricter standards as to what extent they can search a private device, even if they claim the o.s. is only a service.
I wonder what the limits to this are. Clearly if government asks for help then that counts, but what happens if government slowly builds up partnerships where certain 'good partner' behavior is expected but never explicitly required in a contract?
Say government started favoring companies who scan for drug images for any contracts over those who don't? Or say that while working with companies that scan for drug images they build up relationships that lead to both better treatment and to better chances at winning contracts? Maybe companies that refuse to implement the drug abuse material scans end up getting investigated more often.
I wonder how long until the fourth amendment can be considered dead due to this one exception?
Where you do not have both probable cause and a warrant, based on and identifing that cause, signed by a publicly elected judge in the user's (not company/operator's) juridiction.
Preventing a digital platform from governing what kind of content it hosts is just absurd and makes no sense. If apple decides that it does not want to be the platform where CSAM content is distributed then they should be free to do so. If you don’t like being a part of a platform which scans images for abusive content migrate to another platform.
There is significantly more illegal material of greater concern such as communication to plot terrorists attacks or companies ignoring safety regulations and leading to the deaths of many in doing so, and all that isn't scanned for either on the theory that it might exist, not that there is due cause for a search warrant or some similar mechanism and burden.
But child pornography is the one thing Anglo-Saxon culture is notoriously emotional about, and willing to surrender all it's freedom and privacy for. For a while terrorism counted among this too, but that seems to have fallen off.
* If the "target" database is secret, then government entities are free to use a "CSAM" database to monitor dissidents by adding their own items of interest to the target list.
* If the "target" database is public, the only way to validate its contents is to ... traffic in child sex abuse material. That's not great.
Basically, there is no way to create and validate this database in public view. I don't need yet another flavor of secret policing in America.
This doesn't seem like a problem that can be solved without compromising my privacy.
I would rather the problem went un-solved than allow the state to rifle through all of my private files to "prevent the distribution of child sex abuse material," knowing full well the state will define that "material" however they like, then use parallel construction to prosecute dissidents whenever they have gotten information illegally.
It's a privacy-preserving framework to allow the government to monitor the contents of iCloud directly, with few, if any, Apple employees ever having to get their hands dirty.
Apple just handles a target database that gets distributed to phones, and then compiles a list of users whose data had hits against the target database.
Apple employees don't have to dirty their hands with what, perhaps, is in that target database. Not their problem!
This is a complex cryptosystem designed to keep Apple out of the loop. There is a target database whose intended targets they cannot know, and customer data they prefer not to know.
If an account uploads multiple images that match to known exploitative images and exceeds a threshold, then the account is flagged for review by Apple. (Note the threshold is selected to provide a ~1 in 1 trillion probability of incorrectly flagging an account.) Once they review and confirm a match, it's then forwarded to the National Center for Missing & Exploited Children for further action (and presumably referral to Law Enforcement.)
More details in their whitepaper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
The "1 in 1 trillion" figure is accidental flagging on the target database, but there is no validation whatsoever on the target database. How can you, or I, or any other citizen, know whether non-CSAM items are present in the target database?
-------
The NCMEC is a patsy for the police state on this one. It's gross, it's ugly, and it is a terrible outcome for the charity.
In their participation in this program, they make themselves into a front for the CIA, FBI, and DIA forces that are aching for opportunities to crack down on dissent in America. This is an awful, terrible outcome.
--------
The whole thing is an incredibly thin, easily pierced veil for any government. Even if you think the secret police forces of the United States generally do well by citizens, how do you feel about China, or Russia, or Eritrea, or Burma, or Turkmenistan using these tools to flag people trafficking images with undesirable fingerprints?
However, that does not invalidate the fact that apple is in the loop! It’s not just the NCMEC that has to be corrupted - it’s also apple employees. Apple has stated in their whitepaper that they review all flagged content before forwarding to the NCMEC. If the apple employees forward the non-CSAM matches then that is a failure of the reviewers who have betrayed their duty to prevent authoritarian abuse of this system.
1 in 1 trillion is not a particularly impressive number, considering that users have tens of thousands of photos in their library, and there are a billion active iPhones.
That's if you do believe their claimed false positive rate -- which I don't (yet), because they have published absolutely nothing about how neuralMatch actually works. They have an interesting body of work around how suspected matches are encrypted so Apple can't see them until certain conditions are made, but nothing about how they identify suspect photos.
More teachers, smaller classes, stronger relationships, better discipline, more love. I see too many teachers where I work who just don’t care any more.
Educating school leaders to educate their teachers to help children root out the family member that is abusing them. That there would be a pretty neat solution — and I really don’t mean to be inflammatory about this — than catching villains based on photo evidence of the abuse that has become systematic enough to be shared online.
It is very expensive and time consuming though. Teachers are hard to recruit (low pay, low quality job) which exacerbates the problem which makes it even harder to recruit them.
"But the children!" is not a skeleton key for privacy, as far as I'm concerned.
I reject on-device scanning for anything in terms of personal content as a thing that should be done, so, no, I don't have a suggested way to securely accomplish privacy invasions of this nature.
I'm aware that they claim it will only be applied to iCloud based uploads, but I'm also aware that those limits rarely stand the test of governments with gag orders behind them, so if Apple is willing to deploy this functionality, I have to assume that, at some point, it will be used to scan all images on a device, against an ever growing database of "known badness" that cannot be evaluated to find out what's actually in it.
If there existed some way to independently have the database of hashes audited for what was in it, which is a nasty set of problems for images that are illegal to store, and to verify that the database on device only contained things in the canonical store, I might object slightly less, but... even then, the concept of scanning things on my private, encrypted device to identify badness is still incredibly objectionable.
In the battle between privacy and "We can catch all the criminals if we just know more," the government has been collecting huge amounts of data endlessly (see Snowden leaks for details), and yet hasn't proved that this is useful to prevent crimes. Given that, I am absolutely opposed to giving them more data to work with.
I would rather have 10 criminals go free than one innocent person go to prison, and I trust black box algorithms with that as far as I can throw the building they were written in.
Empower children to report abusers and protect them when they do.
I think that's the thing, privacy minded people want solutions that aren't on the spectrum at all.
I can come up with some solutions for greatly reducing CSA that are really low on violating parental rights but they have little chance of gaining support because the only solutions people want are the ones that don't violate parental rights at all (except when parental rights have been suspended due to evidence of CSA being found).
- directly prevents actual abuse of children in the physical world (obviously)
- fights the sale of images created through abuse (as this incentivizes abuse).
Those are the main underlying reasons to fight CSAM, besides the obvious horrificness of the stuff in itself.
Please also consider the second-order effects of an increasingly totalitarian society on the wellbeing of children.
In other words, you don't get to yell think of the children and randsack peoples homes. If you don't have specific evidence suggesting that I am a criminal, you assume that I am not _and_ let me be.
The "privacy" of a on-device scanning is meaningless when you upload to iCloud where they will send to law enforcement and an unaccountable nonprofit as soon as it matches something.
But this will further open possibilities for abuse not only in China but everywhere else as well.
I don't. I've seen what the FBI claims, that they then magically find ways around the "impenetrable encryption" they claim they can't break (but then do), and I don't trust them to not require Apple to add hashes of "things they find problematic," and include a gag order with it so Apple can't report it.
I've worked computer security long enough to know that it's always worse than advertised, always will be abused, etc.
My contention isn't that I trust my government, but instead that this specific technology is no more threatening than the technology already present on my phone and the trust I've already placed in the vendor.
Yes. They could. However, getting caught doing that, from Apple, would be an absolute disaster for them, and they've both made the right statements and made the right actions in terms of things like physical device security (moving more and more into the secure enclave to protect against demonstrated attacks).
It's entirely possible they've done this for specific cases, with a gag order, and nobody knows. However, it cannot be widespread, or it would end up known.
This, however, is adding a blackbox image matching algorithm (we know it's not matching image file data based on statements about how it's robust against cropping/rotation/resizing/etc), and a blackbox database of "Stuff claimed to be Really Bad." There is no way to audit the database and see what's in it, because of the one way nature of hash functions, and there's almost certainly not a pile of the source material laying around for audit reasons (because that would be a really nasty legal problem). So you've got a "Trust us, this stuff is bad" database, and a "Trust us, it matches images accurately" hash function.
And, if those find enough matches (also unspecified), you get yourself reported directly to the various authorities.
It's guilt by algorithm, across every single Apple device (running the latest OS, etc... handwave as needed).
That's very, very different from a customized backdoor deployed to a small number of phones.
The scanner that can be fooled by embedding the images into some document is going to get useless fast. Therefore, PDFs and office formats have to be checked, too. Well, and archives, without doubt, should be unpacked on-the-fly. What if someone embeds the base64-encoded image into HTML? And so on, and so on.
You can't just look at media files, you naturally need to scan everything, and the working thing is likely to be antivirus-like, with many file format filters, heuristics, and extended description language.
Do you have the Collateral Murder video? Abracadabra, and now you don't have it.
Perhaps the community could run a crowdsourced "keep them honest" service web service -- upload the latest illegal-in-China Winnie-the-Pooh meme, oh hey look at that, it's in the China-only version of the database, isn't that weird, etc etc. (Obviously you wouldn't want people "testing" images that are in the database for the actual stated purpose...)
Searching everyone's media for evidence of criminal wrongdoing, but setting only one example of the kind of wrongdoing you're looking for is very susceptible to an actual slippery slope, given that there are plenty of other criminal activities they could start looking for should they decide that it's part of their mission.
The only thing that makes CSAM so attractive to go after is how disgusting society feels that it is. Up next could be reporting drug-related texts of known convicts to parole officers. Next could be drug-related texts of everyone to police officers. Next could be letting the police officers make their own searches, where they find everyone talking about a certain kind of political organizing.
We should refuse to accept even the first instance of this kind of thing.
https://www.txstate.edu/philosophy/resources/fallacy-definit...
In the colloquial language we are using here, there is absolutely evidence, tons of it, that this will take the same downward trend of previous infractions.
Also your characterization of "things change and some of the those end up being bad" is not the nature nor specificity of the evidence in this case.
Lastly, perhaps you should remain consistent and not utilize the "appeal to authority" fallacy.
Second, I'm not appealing to authority. You said a slippery slope isn't a fallacy. I provided a link which makes the case as to why a slippery slope is considered a fallacy.
We are not using "slippery slope" in any way which is different from the examples given.
That one thing has lead to a bad thing in the past is not evidence that this thing will lead to this specific bad thing in the future.
Yes, sometimes, slippery slopes end up being slippery, and sloped.
But that doesn't mean anyone has to take it as a given that any other specific slope will be slippery, nor slippery in some specific way which leads to some specific outcome.
The slippery slope fallacy us real - but so are the slippery slopes.
I remember when an Apple auth server went down and no one could launch non-Apple applications because Apple needed to see the hashes of the things people ran on their computers.[0]
When will the slope be inclined enough for you?!
[0] https://mobile.twitter.com/llanga/status/1326989724704268289
Apple's design was fail closed, so if OCSP is down, assume the application has had its key revoked.
Unfortunately that's just how OCSP is, your browser (if you're using Firefox) does this with CAs. The unfortunate thing is, due to the nature of desktop applications, OCSP stapling doesn't really work when you're not the one serving content.
Uh, sure, fine. But Apple decided they needed to see the hashes of the things people ran on their computers, to possibly block execution if they decide that's necessary, and they don't have to. I'm kinda shocked that you're framing this as if it's innocent, and therefore a bad example of Apple's increasingly paternalistic control. I don't particularly care what sort of RFC they're following or which alternative implementation sucks or which fig-leaf covers their true intent; Apple chose to have the ability to see what people are running in real time. If you're a regular of this forum, and can't immediately imagine how this sort of information might be used to harm users, rather than help them, now or in the future, I don't know what to tell you.
I also don't quite understand how this even helps with the stated goal of virus-corralling. Does the hypothetical virus that they're trying to guard us against change an executable? If so, then the hash is immediately different, but presumably no longer matches its signed checksum, and so could be rejected at the OS level without needing the whole 'real-time seeing what people run' aspect. Does the hypothetical virus run independently? How could it, given the prohibition against non-signed code? I guess the idea is 'prevent a once-legit app from pushing a malicious update and turning several nations worth of Macs into a botnet', style of thing?
You can turn that feature off.
https://en.wikipedia.org/wiki/Online_Certificate_Status_Prot...
They're just following basic pki. Again, Firefox does this with https certs. The CA knows who went to what website.
...uh, then how did the failure of the auth server mean that nothing (except Apple apps) could run? My understanding was that the auth server checked the hash of every application that was being run, and the absence of that auth server meant nothing could run.
Moreover, I don't care about the fact that they're just using plain ol' certs, just like Firefox. They could be using screen recording software and Mechanical Turk to decide whether users can execute some third-party software -- the point is that they are deciding whether users can execute some third-party software. The technical implementation is unimportant, it's a bad thing.
Code revocation checking should be local.
Apple said "Hey everyone owns this U2 album now". And if your phone had automatic downloads turned on it downloaded the album.
The idea that even belongs on the slope is … you're not a serious person and you're not making a serious argument.
It wasn't that simple. The usual functionality of slide-to-delete didn't work at all, because of iCloud, or something. A week after they foisted it on everybody, they decided to implement a dedicated page to allow frustrated users to delete it, because of how different it was from a mere purchase authorization from the iTunes Store. [0]
"The slope", in this case, as far as I understand the root-level ancestor comment, refers to the idea that Apple is increasingly paternalistic about its powers over the device you ostensibly own:
>>>>"We're going to scan your photos, on your encrypted device, to look for badness. Right now, we're going to claim that's only for the really icky people that nobody is going to defend, but, hey, once the tech is in place, who's to say we can't scan for dank memes and stuff?"[1]
I think bringing up the U2 album is entirely within scope re paternalistic overreach. I assume your arbitrary goalpost movement to "invasion of privacy or some new technical advance" is considered by you to be indicative of your personal and argumentative seriousness?
[0] https://www.eonline.com/news/1071481/remember-when-apple-for...
Apple gave everyone the album. It just assigned that album to everyone's library. It could be deleted from the device, but it stayed attached to your account. So it showed up in your library when you searched even if you didn't have it downloaded, because that's how Apple's library worked.
I'm not sure how that's "paternalistic" in any sense of the word.
I didn't move the goalpost. This article, and commentary, is specifically about how Apple is adding technology that can be used to invade our privacy or be abused by state actors.
A bad marketing tactic from seven years ago that amounted to a free gift no one wanted is an absolutely ridiculous example.
The reaction of the police seems to be “why do you care about this? We have the data let’s use it. And we could use it to find those ~killers~ these people that allegedly just killed the most popular crime news reporter. So let us do our work.”
Just like software expands to eat cpu/memory/disk resources and people start living up to their salaries: whatever is possible will happen.
To me the slippery slope argument is perfectly valid.
It's much easier to change the policy than to add a new mechanism. Governments, including the US government have previously attempted to force Apple to break into users' devices, and often apply gag orders to such attempts.
Now they could, and that's enough for me to be unwilling to keep sensitive information on an iOS device.
Someone finding a way to plant this kind of material on your phone does that.
Besides, the fact that something that can be misused exist isn't a very good defense for creating something else that widens the opportunity for abuse significantly.
The blacklist itself is not maintained by Apple, but by the US government or a third party like NCMEC, which means Apple can't be sure content that isn't child abuse imagery hasn't made it onto the list. Perceptual hashes probably can't be abused to target non-image/video content because they're an inherently image-oriented technology.
Apple could, however cause such a program to match on different criteria with a simple update, and such a change would likely be difficult to detect. Most of us assume Apple wouldn't voluntarily do such a thing, but it's very probable that they would do it involuntarily. The US government has already attempted to compel Apple to create a tool to compromise the security of an iPhone, and might have eventually succeeded in court if they hadn't gained access by other means. That fight took place in public, but the next one might well take place in secret.
If you haven't noticed jokes is on you.
It's not subtle. No expansion to the program is necessary to violate your privacy or endanger dissidents.
Who cares what it's for? The fact that they are even doing this at all is a huge infringement to their customer's autonomy.
Moreover, if the database doesn't store actual images and instead only the perceptual hashes, it would be impossible to audit, even if the auditor has access to the database.
All the auditor would see is a bunch of hashes, they wouldn't be able to tell which hash actually represents CP and which hashes represent pictures of confidential documents.
PhotoDNA has been around for a while and you might think about it what you want, but they have never answered those very pertinent questions.
Will they announce an EULA change, reversing the guarantee, and freeze anybody who haven't clicked it through?
I thought Apple was one of a kind company brave enough to put such a guarantee in writing.
And when the FBI said, "Hey, can you write and sign a custom bootloader for this phone to bypass that stuff?" they told the FBI to pound sand and made the hardware security features stronger so even Apple couldn't break them.
And then they bowed to China regarding iCloud and in-country servers, which clearly are accessible to the government. And then this. Whatever claims they've been making about privacy are obviously now crumbling under some external pressure.
On the contrary, if my iPhone really was "my device" then Apple wouldn't make it so hard to sideload software (it was literally impossible to sideload without building software from source yourself until very recently) - and they'd let me change my default browser/maps/contacts/email apps too.
Apple's stance on privacy is something that really doesn't cost them anything but from which they gain massively by using it as a bludgeon against Google and Android (and Facebook...).
The nation state is still supreme- cyberpunk was wrong on that.
https://www.nytimes.com/2021/06/14/podcasts/the-daily/apple-...
You didn't really believe that cheesy open letter, did you? The real reason they declined was because FBI innocently said “We can't bother to transfer phones to Apple all the time, just give us the tool, we've seen how it's done enough times, and can use it just the same. Here's the official demand.” If Apple did that, the next day the tool would be leaked to multiple other agencies and actors, and on the day after any stand on any Chinese market would offer the iPhone unlock and reset for a small price. In other words, the shattering of the legend of privacy protection marketed so hard, and financial trouble. The FBI effectively demanded to hold one of the keys to Apple's power, and was found not significant enough to be esteemed.
Then they all pretended to have a legal fight to smokescreen the public. Apple itself said that it had always cooperated in doing the job on premises, and you can be pretty sure that every official case, semi-official case, and special case leaving no paper trail was routinely accepted. Apple was protecting its own secrets, not yours.
Given this, Apple was not a bad person to trust. Yes they wanted you to pay them money, but were a) quite happy to deliver value in return, b) prepared to do lots of hard work to fight on your behalf and c) it was quite clear that you were their customer, not their product. Google is an advertising company, even if you pay them money you are always also the product.
I am seriously wondering what happened to Apple that they came up with this idea, it doesn't seem to be in their interests at all.
Which markets are closed to Apple by its own government currently? North Korea?
Apple has bowed to them on iCloud server locations and access already.
From the link: "Before an image is stored in iCloud Photos..."
This leads me to believe that only data that is going to get uploaded to their servers is going to be scanned. If anyone has a different interpretation or thinks I'm wrong, feel free to reply.
It's my understanding that all cloud services do this type of scanning, when they are technically able to.
But all data on iCloud is encrypted by default, so Apple can't scan for this kind of material once it is on their servers. Doing it on device before it gets encrypted and uploaded is the only place they even could do a scan like this.
Additionally, they make it clear in the article that there has to be more than one hit (they don't say the actual number) which would mitigate risk of hash collision false positives.
If this type of scanning makes you uncomfortable, you can just not use their cloud services.
I do agree that this is still not a good direction to go, even with all the precautions they've taken. But I had to do some digging to figure out what was actually going on, the comments/commentary made it seem like Apple is now routinely scanning all your photos/videos if you have an iPhone.
Once the code is there to do local scanning, it might make it easier for a zero day exploit to do phone scanning and grab data it might not otherwise have access to or for governments to force Apple to conduct scans of content on a phone when they ask.
I guess the point I am making is that as of now, this only applies if you're using their cloud services. I'm not sure if Apple would announce if they were compelled to use this functionality through a court order.
They also confirmed it is only when the photos upload to iCloud: https://www.macrumors.com/2021/08/05/apple-csam-detection-di...
If you allow me to guesstimate wildly, most “child porn” these days, in a technical sense, is made by kids themselves having access to an internet-connected device with a camera. Sometimes it is extorted by despicable abusers, sometimes it is done for no one in particular, just for the perceived fame/popularity/likes on social services/etc. Big services have an army of moderating grunts to keep Victorian purity of blissfully ignorant common public intact, things are a bit different in poorer parts on the Net and on the Globe. Should we expect the naked selfie of a teenager sent to their significant other to automatically cause a police interrogation of the same teenager? What if the device is shared with older members of the family? Another man-made dystopia enabled by people who enjoy to express outrage over the racy stories in the media, and people who are too afraid to speak up.
In essence, Apple has introduced a software agent to signal whether you have files that belong the list someone provides. If I recall the scandal correctly, this is what Kaspersky allegedly (ab)used, and what other antivirus tools (including Microsoft's built-in and enabled-by-default Windows scanner that for some seemingly important reason nags all the time if uploading of files to Microsoft is disabled) surely enjoy offering to various agencies around the world.
I don't think you should worry about “China, Australia, and Saudi Arabia” so much, there's elephant in the room you don't like to notice.
It looks like it only triggers when you go to upload a photo to iCloud. In that case, it (maybe?) scans only that photo on your device and then decides if it's CSAM.
I guess Apple really really doesn't want CSAM on their servers? I don't know why they wouldn't just perform the scanning on their own server otherwise.
iCloud stuff already isn’t E2E and they already scan.
This system is built exactly to surveil the entire contents of your mobile device, cloud enabled or not.
Apple has burned years of social capital in 2 days.
It's likely that this tech originated from Apples capitulation to China and has been in place. So it's probably been in place for quite some time.
I see it everywhere and it literally seems like some kind of one-way entropic process. I can't think of anything that would reverse it. It would be like turning an omelet into an egg.
Is there something about modern life that just inescapably creates this complacent, servile temperament in most of the population? Or has it always been there and I'm just overthinking it? It's really depressing either way so I try not to think about it
Thomas Jefferson had some arboriculture advice that seems relevant to this kind of thing.
I do not think any of the founders, however amorphously the term is applied, would have interesting thoughts about cryptosystems for the examination of private photographs stored in commercial cloud setups.
I only have anecdotal evidence, but it seems like most people don't care about problems unless they affect themselves personally right now. Also, most of us (HN aside) don't tend to think seriously about how things will go wrong, we're generally optimists that give more weight to the good something will do.
The "This could be sensitive to view" screen is downright Orwellian. This technology could be used to scan for ANYTHING, completely undermining user privacy. It might just be CP today, but tomorrow it could be screenshots of protest material, whistleblower content, or anti-government memes.
I cannot express how sad I am Apple has decided to do this. It doesn't protect children, it won't catch any pedophiles, but it certainly WILL be misused in the future and create a chilling effect on what (politically dissident) content people are willing to store on their phones.
Scary times; for exactly the reasons you've called out.
I can’t find an extended explanation right now, but they say as much under the photos heading on their privacy page: https://www.apple.com/privacy/
#!/bin/sh
#epub.sh
unzip -qcp "$1" "*ml" "*.htm" | lynx -force-html -dump -stdin -nolist > "$2"
MP3 player, simple HTML browser, FM radio (it's nice often), Gopher (there are a few J2ME browsers, and there are lots of nice stuff out there). For gaming, IF thru the T9 kbd it's effective, or a GB emulator with custom ROMs (itch.io has several free and a lot of games are long and turn based).
Use a smartphone for work and for nothing else. If any, the news over a dumb web interface, as these will run on current dumbphone with ease:
Have fun.
The gradual but steadily accelerating rise of authoritarianism scares me far more than terrorists, drugs, child abuse, and the pandemic.
Unless we push back mightily, it will be a question of when, not if, owning a general-purpose computer that's not controlled by the government or a company becomes discouraged, suspicious, and eventually illegal.
Companies holding interests in copyrighted works would love to see general purpose computers go away, replaced by "trusted" media players that can't make unauthorized copies or be used to make unsanctioned independent creative works.
Totalitarian governments would love to take away general purpose computers to prevent end users from removing surveillance and anonymity.
Companies who want to control software markets would love it if all software licensing transactions ran thru their "marketplace".
Of course, his suggested solution of open-source didn't quite work out --- because even with the source available, you're still enslaved by things like Secure Boot and other cryptographic jails; and on the other hand, as evidenced by the long history of the cracking scene, and Windows modding in general, not having the source is no big obstacle as long as you can still arbitrarily change any bit. Perhaps what is truly important to software freedom is not "right to read (the source)", but "right to write (anything)".
"Oh you run linux? Do you not trust the government, or are you trying to hide something?"
Yes, it's all "they could" but with current technical solutions already providing some measure of protection against a corrupt or malicious government from cracking down on its citizens, anything that erodes from that freedom deserves to be held to such a high standard.
Apple has no visibility into the original image that generated the hash, so the gov can compromise the list at the source and Apple would have plausible deniability.
https://en.wikipedia.org/wiki/John_Walsh_(television_host)#C...
There is literally no oversight here. It's a misfeature and will be misused.
Can’t say I‘m surprised.
I know a local family who has a daughter who's been in therapy for the last 3 years because she fell victim to the type of thing Apple is discussing in this post. They are firmly advocates for better parent education and oversight, sharing their experience so that other people can hopefully never have to deal with the same thing. They told us about an app called Bark[1] that's supposed to really help with a lot of this stuff and seems inline with what Apple is talking about here. I'm pretty happy to see it will be built in.
> The Messages app will use on-device machine learning to warn about sensitive content, while keeping private communications unreadable by Apple.
All the parental controls in the world don't prevent the fact that getting your kids a phone in this day and age is a pretty terrifying experience if you know what type of things are out there.
> When receiving this type of content, the photo will be blurred and the child will be warned, presented with helpful resources, and reassured it is okay if they do not want to view this photo. As an additional precaution, the child can also be told that, to make sure they are safe, their parents will get a message if they do view it. Similar protections are available if a child attempts to send sexually explicit photos. The child will be warned before the photo is sent, and the parents can receive a message if the child chooses to send it.
there is a really easy solution to this problem
a) Many places are still going in/out of lockdown or are schooling remotely and so their phone is the only way for them to communicate with their friends. Depriving them of social contact is incredibly unhealthy and harms their development.
b) For better or worse apps like Tiktok are a huge part of their culture and the popular dances etc are often known by everyone. Being the only child who is out of the loop can cause serious isolation.
Children are growing and making them not feel like they are part of a social group is incredibly harmful and can have permanent effects in adulthood. Giving them a phone but monitoring their activities is likely to be the least harmful approach.
Your line of thinking is ridiculous but please continue.
It sounds like there's a second, unrelated thing going on which cannot be turned-off that reports images to apple that set off their warnings. The large concern there is that this type of techology could obviously someday be used to say report/delete all photos of police brutality. Since 20% of the world alone lives in China, I think the question of "How do we ensure a malicious authorities cannot use this technology against good users" is not really an afterthought but must be addressed head-on before people will buy-in.
Given how free US Law enforcement is with violence, any potential threat of involvement with them makes me very very nervous.
Going to need to reconsider hosting my photos with Apple devices.
Also, it looks like the blurring feature is limited to the Messages app. That's pretty easy to work around.
Additionally, the client side scanning seems very well-designed, but if iCloud Photos are not end-to-end encrypted, why are they going to such an effort to do this when they already have access to any image they want server-side?
https://www.apple.com/child-safety/pdf/Technical_Assessment_...
Also look at their full whitepaper here: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
My take is that performing the initial hash matching and encrypting the results in two separate layers on device prevents Apple from having meaningful knowledge of low (under the set threshold for being flagged) quantities of matches on a user account. This protects the use of the threshold as a way to further reduce false positives. For example they couldn't comply with a subpoena that said "Hey, we know you set a threshold of only flagging + reporting accounts with 50 image matches, but we want to see a list of all accounts with 10 or more matches because we think that's good enough."
This method lets them set and enforce a threshold to maintain their target false positive rate which they say is ~1 in 1 trillion accounts incorrectly flagged.
Disclaimer: I'm not a cryptographer and could be misunderstanding this.
The only remotely plausible answer I've seen is that Apple wants to keep potentially-violating material out of their general storage, and flagged images are being sent to the review team instead of regular backup, but that's a pretty weak guess.
https://www.apple.com/child-safety/pdf/Technical_Assessment_...
Apple's official proof was done in part by Dan Boneh:
https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...
It is great.
But it didn't prevent Facebook from trying to abuse WhatsApp users in every conceivable other way including some rather innovative ones as seen recently.
Also it didn't prevent Signal from releasing a desktop client with a really nasty XSS bug and the phone client hasn't exactly been without faults either.
You and Moxie are both much smarter than me but I foresaw and maybe even predicted (possibly so early it is under another handle) some of Facebooks actions while everyone else was just talking about how E2E-encryption was the big difference.
This feels more or less the same: the intent is honorable, the core code is clever and almost unbreakable - but you cannot really trust all of the actors you need to trust for the system to work.
We cannot read WhatsApp messages in transit. And, as specified this system sounds good. But WhatsApp happily uploads unencrypted copies to Google Cloud and/or iCloud without your consent if another participant enable backups, and I'm fairly sure we'll see this system containing huge opportunities for abuse by power hungry regimes as well as one or two nasty bugs that has the possibility to utterly destroy a few innocent persons lives.
And right now, I don’t trust the maintainers of the CSAM database and Apple to do the right thing.
Nobody signed up/purchased an iPhone trusting random 3rd parties/CSAM aggregators.
These brilliant academics just don't care so long as they can publish an interesting paper I guess. Whatever, still reading.
2. They get to do some interesting work.
Those are the positives for them I see.
Rather than abstaining they work on it and lend their names. I don't know whether they support what Apple is doing with the protocol or if they just like figuring out the possibilities. I don't think either is good in this case because I don't like the very first application of this PSI System. Maybe I'm being short sighted and jumping to conclusions all at once. I still don't like it.
Honestly, any time there is a new policy to "protect children" it is almost always incredibly invasive and it always feels like there is some other motive and "protecting children" is used to scare anyone who tries to question it.
I’m also struggling to imagine scenarios where a child predator is clever enough to acquire illegal photos without triggering any number of internet monitoring mechanisms (e.g. honeypots, server logs with their IP address) who would then turn around and upload those photos to their iCloud account. Doesn’t make sense.
This is a really strange move.
why not just run the scans in the background... (honestly surprised they aren't already) it's not like it would be hard to omit this avenue as your lead during prosecution
I just spent the last hour of my life digging through the material. It seems like they've calibrated the system to have an expected false classification rate of one account in one trillion, per year. Based on the threshold secret sharing math, they won't know anything until a user has a significant collection of CSAM and the secret key is recovered.
> I’m also struggling to imagine scenarios where a child predator is clever enough to acquire illegal photos
I don't want to be the bearer of bad news, but CSAM has been shared openly on the clearnet on places like 4chan for years. The internet is a pretty wide open search space. Many of the people who download CSAM don't do so from some sketchy underground website. This is why Tumblr, for instance, burned their platform to the ground: lots of people sharing CSAM with no way to detect and stop it (without costing Verizon a ton of money to work on the problem).
My guess is that this is a PR campaign to show that they're doing self-regulation. Perhaps related to the EU push against end-to-end encryption.
It's the part about scanning people's photo libraries that folks are (rightly) concerned about.
The more platforms we make unable to keep CSAM on disk, the harder it is for it to circulate in a viral fashion. With this change, all iOS machines become immediately unviable as CP transmission vectors.
The fact that it's only targeting users doing something almost universally considered heinous is irrelevant. If Apple can scan your encrypted photos to make sure they don't contain child abuse, why not scan your encrypted chats to make sure they don't contain a conspiracy to commit murder, or an arrangement to buy drugs, or a plan to reveal something embarrassing to the ruling party?
People often posted messages containing these Unicode characters in Discord groups (tagging @everyone) for the lulz, because every iPhone user in that group would get a notification containing those unicode characters and then kernel panic.
It's only a matter of time until someone finds a bunch of false positives, spams them around for the lulz, and boom people's iCloud accounts are disabled.
Even if you believe the theory is mathematically sound, the implementation need not be.
The idea that apple can scan files on my system without my consent is pretty sickening. I don't care what its purportedly used for. This is a slippery slope to all sorts of privacy violations.
Do you not own your device because apple puts iOS on your iphone before you buy it?
On-device scanning of “encrypted” cloud content where Apple has always held the key makes 0 sense unless your plan is total device surveillance.
They're doing this because they are announcing it, because they think it's a net positive. You might not agree, but the argument that this is somehow creating the technology for scanning and is therefore nefarious is missing the point that if they wanted to do this surreptitiously and nefariously, they (a) wouldn't announce it and (b) could have been doing it for years. This isn't some fancy new tech, except for the privacy bits, which obviously wouldn't apply to Evil Schemes.
[0] https://www.zerohedge.com/technology/apple-plans-monitor-all...
The reporting isn't helping, with sentences like 'the company is rolling out a new machine-learning tool that will scan iPhones for images that match certain "perceptual hashes"'... is that describing a system that classifies new photos, or is it talking about comparing hashes to a known-evil set?
- Only applies to photos uploaded to iCloud
- Matching against a known set of CSAM (Child Sexual Abuse Material) hashes occurs on-device (as opposed to the on-server matching done by many other providers)
- Multiple matches (unspecified threshold) are required to trigger a manual review of matched photos and potential account suspension
There is also scanning for nudity in the Messages app, but those scans happen on-device and the photos stay on-device even if nudity is detected.
[1] https://www.telegraph.co.uk/technology/2020/01/08/apple-scan... [2] https://web.archive.org/web/20200110193302/https://www.apple...
I have no words.
This will be used in Xinjiang and Hong Kong. And soon, it will be used against you.
The only relief I have is that Millennials and GenZ aren't having kids and many couldn't give two fucks about them "being protected" from imagined goblins at a cost to their convenience. Maybe this argument won't work anymore.
Have we already forgotten about where fear of terrorism has landed us?
I would also not celebrate random warrantless house searches even if it managed to stop a couple of terrorists attacks in a decade.
Anyone motivated to engage in that kind of behavior is just going to use a different device for their illicit material.
Perhaps this same approach can also soon be applied to filtering out the lies and misinformation of the enemies of the Party and to disrupt Emmanuel Goldstein's networks of treasonous domestic terrorists. After all, such misinformation is a plain harm and danger to IngSoc. That Apple is willing to eliminate nudity in private messages gives hope that the Party will soon be able to free all society from doubleplusungood differenthink.
They will pop a warning when the minor sender attempts to send something that is flagged as nudity, or the minor receiver receives something that is, if enabled as part of parental controls for the account.
It's opt-in for the account.
The CSAM scanning is different, unrelated to the age of account holder, is a condition of iCloud Photos usage only, and is not opt-in (and the only opt-out is to not use iCloud Photos).
As an obvious vector - an abusive husband can set up the account on their partners phone, set them up with a child account and "check" they are not sending anything suggestive. And we have no idea how reliable the scanning is for the sexting piece. A false positive notification could easily get some beaten, even possibly killed.
Scanning against a known database of material uploaded to iCloud didn't sound too terrible for me, but this sexting feature is incredibly dangerous.
No official source is protecting this one from users flagging it off the front page.
If you're on the fence about Apple, and their lockdown of free and open computing wasn't enough, or their strong arming of the mobile commerce economy didn't do it, please let this sink in. This is where we're headed. 1984 is a plausible future.
Stop buying Apple and ask your representatives to break up the monopolies. This order came from the FBI and intelligence communities (and probably the CCP and respective organs in China).
Not all US lawmakers are beholden to this abuse of power, and they can help us put an end to it. Call them. And stop buying Apple.
Fight for your democracy and freedom, lest it be completely wrested from us.
Repressive regime TODO list:
1: create a child safety organization, or require an existing one to accept your images
2: add images of the children of dissidents (or journalists, or leaders of other political parties), photoshopped to be sexually explicit
3: dissident iphones informs on them. Apple turns the information over to the authorities in the host country
4: if Apple pushes back, threaten iphone sales. Or just improve your doctoring.
5: if Apple plays along or doesn't complain, insist on the ability to detect terrorists, criminals, etc. Again, threaten iphone sales, allow Apple to keep the agreement secret.
This may only work once or twice, but it's worth a shot! If you make it to step 5, you have a really bespoke, beautifully designed, Apple managed intelligence apparatus. made with love in Cupertino.
Maybe this is an overly cynical take, and in addition to the cryptography they have rock-solid, audited governance and internal controls that would prevent it and/or insider abuse.
Maybe localities with real data privacy laws (EU) will be able to offer protections to their citizens with fines big enough Apple will begrudgingly agree, so that a repressive regime can't target their citizens as well as citizens in the host country.
Maybe this isn't a slippery slope to more exotic forms of surveillance, like scanning your contact list for pedophiles.
"additional encrypted data" makes it sound like it's encrypted to keep your information safe, when it actually means "encrypted so you can't see what's being sent" and that "only Apple (and people/governments Apple shares decryption keys with) can decrypt it." And what does "additional encrypted data" even mean? I'm guessing it includes a thumbnail. It could include a list of people you shared the image with and their home addresses. The point is, you don't (and can't) know.
Not only that I do not consent to using the device I purchased for such measures.
I need a grand reset to 2005 levels of technology at this point.
Terrifying. This will be used in other ways. Whenever you hear "protect the children" as an excuse to increase surveillance you know they're up to something horrible. The future is bleak AF.
This is absolutely going to lead to deeper use of this kind of anti-encryption tech. third party doctrine loophole to the 4th amendment -- pretty soon every document on your PC will be crosschecked for a litany of things the government deems illegal.
> allow Apple to detect known CSAM images stored in iCloud Photos. ... Apple only learns about users’ photos if they have a collection of known CSAM in their iCloud Photos account.
The only non-cloud hosted scan is for child accounts, with Apple getting no access, just transferring the warning to the parent.
Prove that. Release all git commits and emails and communication and who was involved in these features. I don't see any reason they would not do that, if this was about "child safety".
My guess is a portion of these "child safety experts" will have emails ending in "nsa.gov".
Are they saying that they physically can’t access your iCloud until the threshold is reached, or just that they “promise not to”?
> iCloud content may include email, stored photos, documents, contacts, calendars, bookmarks, Safari Browsing History, Maps Search History, Messages and iOS device backups. iOS device backups may include photos and videos in the Camera Roll, device settings, app data, iMessage, Business Chat, SMS, and MMS messages and voicemail. All iCloud content data stored by Apple is encrypted at the location of the server. When third-party vendors are used to store data, Apple never gives them the encryption keys. Apple retains the encryption keys in its U.S. data centers. iCloud content, as it exists in the customer’s account, may be provided in response to a search warrant issued upon a showing of probable cause, or customer consent.
This happens to thousands of accounts every year.[2]
1. See page 11: https://www.apple.com/legal/privacy/law-enforcement-guidelin...
Various governments, on the other hand, have been very anti-encryption over the past couple decades. Australia's anti-encryption law[0] is just one example but I'm sure there are many others.
This presents an ongoing threat to Apple's current strategy.
The most common arguments used by governments to justify anti-encryption legislation are child protection and anti-terrorism.
I wonder if this is a tactic by Apple to undermine that common argument and pro-actively protect their rights to employ end-to-end encryption and other privacy features.
[0] https://fee.org/articles/australia-s-unprecedented-encryptio...
The accuracy of not just the detection rate (there are some outlandish claims of once in billion here), but also the accuracy of the NCMEC database are really the main concern, as well as Apple keeping this system limited to this specific scope.
Interesting aside: I once attempted to get access to PhotoDNA, essentially the only insurance against malicious actors abusing upload fields on your website to "digitally swat" you (as has happened to a twitch streamer with an open Dropbox folder), and there is no way you'll get access without a department of lawyers. Why is NCMEC is so protective of an API with rate limits and automated reporting features and then would let Apple ship a bloom filter.
All that's needed is a law prohibiting unsecured computing devices by government to plugin any 'gaps' (cue: non tpm, locked down bootloader devices)
I don’t use iCloud (only for MS MFA backups), is this affecting me?
I self host everything, what happens when my account is suspended, does my iPhone work?
I sure hope we get to learn how effective this has been in say, a year? It’s quite something that my device is going to scan and check my pictures. I’d really like there to be a large, confirmed benefit for the children.
1) its only scanned on upload to iCloud, so if you don't upload then its not scanned
2) (per another article, https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni...): Most cloud services — Dropbox, Google, and Microsoft to name a few — already scan user files for content that might violate their terms of service or be potentially illegal, like CSAM.
So you really can't opt out unless you avoid all cloud photos
Hard to make any absolute claims about how it works when they admit it will be changing.
Reading through the two linked PDF's, I got the impression that they're aiming to use cryptographic techniques to meet a (possibly self-imposed) requirement to scan for CSAM while revoking their current ability to decrypt photos stored in iCloud. I guess they may want to revoke this ability so that they can no longer be compelled to hand over customer's photos except those for which the requirements of this new system are met.
I guess if it results in an increase of the privacy of iCloud content, that's nice as far as it goes. But it does skeeve me out to be reminded that my phone can (and does) paw through my stuff.
"Apple then manually reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC" -- So doesn't "reviewing reports" in this hypothetical mean looking at a child's nudes without consents in this case? The documentation left out the part where this file, stored entirely locally, gets uploaded to apple. Or does this only happen for iCloud?
Also, is this all configured on the server-side? Suppose Apple has a pervy employee who vouches my daughter's account "looks suspicious," would he/she then be able to override or "manually review" all her pictures? Would I be notified of any such review?
There are a lot of unaddressed questions here.
1. Nobody new can see images sent over iMessage under the proposed changes.
2. A perverted employee cannot flag an account as suspicious; only a device can flag itself when a threshold number of photos matching the NCMEC database are found.
3. Accounts which meet the threshold and have the photo thumbnail verified as objectionable will be suspended with an opportunity to appeal.
Take for example point 2. According to what they've written here, yes that's how it works. But the point is, you have no flipping clue what's actually true, just what they typed. There could be debug modes, the false-positive numbers could be entirely incorrect, they never said what the "threshhold number" is, so that could be 1. They could have even meant everything that they said, but there could be a bug in the code.
Suppose there is a bug they discover, and the false-positives are exponentially higher than they quoted. Do you think they would publicly admit that? Also how big are these thumbnails and under what circumstances do they have access to them?
iCloud was already scanning for CSAM, this time though they do it without seeing what you upload to the server.
This is essentially encryption with a backdoor, and the chief objection is basically the same as the Clipper chip/Skipjack. There are privacy issues with the idea of backdoored encryption, as the EFF pointed out. To counter misinformation that is being spread, it's not a little AI cop doing blanket scans on your image library and app caches - it is expressly intended as a part of iCloud CSAM detection.
But at the same time it seems trivial for Apple to expand it to that someday, since they just deployed a method capable of doing so trivially. Conceivably they can make the API so that it does a scan when an image is saved to disk. This is not currently what will be done but whether or not they will is anyone's guess.
I will be pushing for our current & future customers to use alternative device vendors as part of our product stack. We have developed a cross platform implementation of our iOS application recently, and this is a great opportunity to start putting it to use.
I don't know if Apple is paying attention, but there are enterprise customers out there paying them $300/yr for the permission to use their presumably-secure devices for purposes of running custom LOB applications. Our organization and product are part of the reason you have financial institutions making bulk purchases of iPads and other Apple accessories.
If Apple makes it impossible for us to provide assurances to our customers that no data is leaving the devices (aside from agreed-upon application protocols), then we have a serious problem. I am not putting myself in a position where I push a customer to purchase 1000+ iPads for their business if I will then have to answer to their auditors 6 months later on when sensitive customer documents hashes or whatever other horrible things start getting sent to the mothership as part of v2.0 of this shitty idea. Apple even said it themselves - "These efforts will evolve and expand over time.".
Is there a special order form if I need to purchase a non-bugged iPad for a classified government setting or very pissed-off enterprise customer? If not, our default recommendation is going to be Microsoft Surface and we will totally deprecate usage of Apple hardware over the course of the next 18-24 months. This is setting a completely unacceptable precedent, and most of our customers I have spoken with today also agree.
In the US weed is illegal at a federal level, so we can get that as well.
Woo!
And does anyone think this will actually catch any fucking pedos?
Oh, joy, if a robotic bureaucrat nukes my personal data, I have the option to file an appeal that will probably never be looked at.
Also this is US only? What about other children?
The US also has stronger laws so this is also about protecting Apple.
The UK ISPs implemented Cleanfeed, a joint database to block access to child abuse materials, many years ago. Your feelings were pretty common.
Since then Cleanfeed has expanded to censor copyright infringing sites by court diktact, and proposed legislation (which will, in some form pass) will effectively use it to block all sites not suitable for under 13 unless they implement age verification and where the guidelines can be updated at will explicitly for policial purposes by the Secretary of State.
So it is pretty hard to take your statement seriously where such slope exists and many of us are much further down it.
Have I missed something?
Running a porn site with underage content and receiving big money from big payment processing systems stopped being an option in the early 2000s after some big busts. Then individuals would try it, and would try to use shady payment sites linked to other crimes. Then child porn, and even traces of past discussions of it, simply disappeared from open web (thanks to Google silently sugarcoating the search results, and hosting and ad businesses being increasingly picky about what they want to see). A typical person won't be able to find any of those producers (apart from obvious scams and honeypots) neither on open web nor on anonymous networks. Heck, even I, having some experience, probably won't be able to find anyone “selling child porn” even if I was told to do so and visit every possible place of discussion.
Though I agree that producing all kinds of media with modern technology got easier enormously. A dedicated handheld camera looks like a steam engine today.
At the very least, this makes retroactive scans of a newly banned content difficult without explicit deployment by Apple, and explicit software update from a user. In Western countries, this friction is enough to ensure people will be able to file suit in courts to block the government.
In authoritarian governments, it's not, but then, nothing is.
Any government of a country with a large enough market for Apple, can force Apple to include hashes hashes into localized versions of OS.
What happens when lawmakers make this mandatory? In such a way where the user can't be allowed to remove it?
If that ever happens, you've just outlawed phones/computers that allow their users to have ultimate control over the software running on them. No more alternative desktop/mobile OSes.
Oops.
For example would it be easy now to get to the hypothetical scenario where a text containing certain phrases will be flagged if some partner / regulator demands that?
The fact that they raised the children card speaks volumes about the level of fuckery they're gonna deploy on this one.
It all may still end up being a slippery slope; but with this perhaps the chance is slightly lower (than if Apple introduced these changes without as much of a comment).
> Messages uses on-device machine learning to analyze image attachments and determine if a photo is sexually explicit. The feature is designed so that Apple does not get access to the messages.
Almost every government blocking system started for the sake of the children, then expanded for this or that reason. In some cases, it expanded before starting: notoriously, first sites blocked when Russian government censorship was introduced (to protect our kids from the dangers of the internet!) were well known media entities that had criticized Putin for a long time. The zest was that they were hurriedly blocked even before the relevant acts came into effect. Like, why keep the act after the deed is done? So now there is a government agency that has been blocking oh-so-edgy pictures of nooses on DeviantArt and similar content for years to pretend that they care about the children, and not execute censorship (and sell their extortion services to people who want to remove some information from the Web on the side). Whether that helped any kid is an open question.
Send or otherwise inject problematic images to the target, but make sure the target can't see it (image too small, part of larger collage), but their device does. The target gets flagged by Apple for having CSAM.
The target gets FBI'd.
Edit: Minor correction, the scanning on your photos does happen on-device but only if you are going to upload it to iCloud.
As long as you stay on iOS 14/iPadOS 14/watchOS 7/Big Sur then you will never get the on-device features. The photo scanning is happening on photos you upload to iCloud. Other than for malware I'm not aware of Apple ever pushing an OTA (in the sense that the user doesn't get to choose to install the update since almost all updates are technically "OTA"). Even the most recent Pegasus hack fix is in an update you can choose to install.
I consider iOS to be the best platform for kids today. That said, Apple: if you're listening: please tighten up parental controls around time limits and re-loading apps!
How about when Apple gets a NSL or other non-disclosable warrant and searches my phone with this tool? Still no?
Probably going to be a lot of leaked nudes by Apple soon.
Sure, they're only scanning for images right now. But there's nothing stopping them from scanning for other things, like Mein Kampf or the writings of Karl Marx. Or some other author who suddenly finds themselves unpopular with the government of the day.
This constant yo-yo of privacy marketing and bad privacy decisions is too time consuming and mentally stressful.
I was waiting to update my 2017 MBP to a M1 16" when they come out and when my iphone 11 is coming up 2 years. These products will no longer be apple products and idgaf if I have to deal with linux desktop and jailbreaking an android to get it done.
A sad say for privacy and freedom.
Obviously any child porn owners will stop using Apple devices, while the rest of us suffer the consequences.
I always wonder what would happen if Tim Cook will retire one day and be replaced by someone like Donald Trump. Once all the tools are in place...
Your actions or lack thereof will have grave consequences for billions of people across generations to come.
However I think this is an interesting question: how does Apple know that the hashes they’re supplied match CSAM, and not, say, anti-government material? How would they know if the people they got hashes from started supplying anti-government hashes? Apple will only be receiving the hashes here - by design, even they won’t have access to the underlying content to verify what the hashes are for.
Increases the worry for me. I want to be invest in a company that cares for property rights.
No human could possibly be unable to correctly identify an image of child porn, and someday algorithms will reach that point too. And once they do....it's probably not a big leap from there for browsers and operating systems to start denying the images altogether, just blacking them out and preventing their transmission over networks.
The flipside is that it is impossible for even a human to tell the difference between a 17yo and an 18yo, and moreover impossible for a computer (and arguably for a human) to know whether the user of a device is sending images of themself (i.e. a 17yo sexting with their bf/gf) or whether it's exploitation. That's harder.
So for like this new post from apple, this is going to be pretty shitty for high schoolers trying to sext their boyfriends and girlfriends.
I run GNU/Linux, and use encryption. Any mandatory technology which could scan my hard drive for child porn could scan it for:
- Antigovernment subversive materials
- Embarrassing things to blackmail me with
- Non-PC communications
- Right-wing/left-wing political materials
- And so on
I'd like to continue to have a computer I control, and I'd like my child to grow up with that freedom too. I'm much more worried about risks to my child's freedom and privacy than the (exceptionally rare but frightening) chance of exploitation for child pornography.
As a footnote, many algorithms able to correctly identify child porn would also be able to generate it.