Apple plans to scan US iPhones for child abuse imagery
ft.com
ft.com
I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.
I am not sure the accuracy of these systems but even if they are pretty accurate there is something off putting about this trend.
This very much feels like a Guilty until proven innocent type of program... "if you have nothing to hide you have nothing to fear"...
Neither of those normally work out really well.
But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I agree though, the scary part might be how to clear your name after an overreaction occurs, similar to the effect swatting has on innocents or other kinds of false accusations. But those would happen even without such a system - we’ve heard about such stories before. “Man held for child porn charges, but it turned out they were innocent,” - it doesn’t take this kind of system to have police look over your phone for evidence and find a different crime than expected.
That all said… child porn is really, really terrible and I’m actually in favour of this. As a society, we can decide on what’s acceptable. As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes. I do agree though, all these systems might make it harder to catch perpetrators if it pushes them offline. But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I suppose the fundamental problem is that having more evidence doesn’t necessarily lead to justice if it paints a false picture.
This also limits the possible avenues for future exposure - if one has less ways to distribute or get CP or the overall amount of it is reduced, they may never get into it in the first place and the market for it will shrink. Thereby reducing the amount produced over time.
This is a horrible idea, a massive regression in Apple (and therefore the broader industry's) stance on security/privacy, and the fact that you are dutifully swallowing the "child porn" justification is so incredibly concerning to me.
You're also being naive by swallowing the "it's only iCloud" justification. There is a clear intent to progress to scanning all local photos. After all, if photos must be match those of a government child abuse database, and any matches trigger manual review, what would an innocent person possibly have to hide? By that logic it'd be absurd to not scan just because it's only locally. After all, you're only catching bad guys! This is about protecting the children after all! /s
I oppose government-supplied blacklists. Doubly so when we can't even see what's in the database, but I would oppose them either way.
I don’t appreciate you calling me naive. There isn’t clear intent to scanning all local photos. In fact, it seems like the system was deliberately designed to only work with photos that are being uploaded to iCloud.
When someone is trying to rob you, generally the best thing to do is to hand over your wallet. If a law passes you should obey it for your own sake. That's an entirely orthogonal question to whether we as the users should be outraged or not.
Also while obeying an injust law, a company has a responsibility to legally challenge it in court with their resources, if for no other reason than to avoid the profit loss of having to comply with invasive regulations
Furthermore, it's not clear that this is the direct result of some federal law. It reads more like pro-actively appeasing their federal overlords.
It is good advice not to carry around huge amounts of money on your wallet. Someone might want it and have the power to get it.
Same with megacorps constructing and bragging about tools for oppression:
Better not do it.
Otherwise just be prepared that China will demand that you also upload their hashes of extremely bad photos - and one or two guys they want to get hold of.
It is kind of a Manhattan project for digital privacy - it is meant to bring a quick end to a actual real problem we have - but it has the potential to become a shadow over society for decades if not centuries to come.
I doubt they need, or want to match more than one. They'll use the reasoning that it only catches already-known child pornography, and that that person might have a whole cache of novel child pornography. Just look at the drug wars and how many people get arrested for trivial quantities of marijuana. And we hate child pornography way more than drugs.
> As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes.
The judicial system has a presumption of innocence. No one else does, or has any requirement to. The court of public opinion will ruin your life regardless of whether it's true or not.
That would be far less concerning if active cases were sealed, and only published if the defendant was ruled guilty.
> That all said… child porn is really, really terrible and I’m actually in favour of this.
I have a slight disagreement with this. The production of child pornography is obviously abhorrent. This system does nothing to stop that though, because it can only flag already-known child pornography.
This system stops the consumption of already produced and documented child pornography. While it's far outside my own interests, I can see some validity to the argument that consumption is of far less concern because it doesn't directly involve any abuse. The abuse has already happened, and no amount of tracking down pedophiles is going to reverse that. There are second order effects, like potentially reducing the demand for novel child pornography, though. Which leads me to this:
> But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I would imagine this is far more likely to affect consumers than producers. In the drug market, stop and frisk is far more likely to net you users than producers or dealers. For one, there are more users than dealers/producers. For two, dealers, but particularly producers, are typically far more cautious than users.
That's a long-winded way of saying again that this doesn't seem likely to do much in the way of stopping people from abusing children.
But to directly answer your question: doing nothing is absolutely a valid option here. We do that for lots of bad things that happen when the costs of the solution are unbearable. It's what we're currently doing for famine in other parts of the world, which almost surely impacts more children than child pornography could even dream of doing. It's what we do for cars, which again, almost certainly impacts more children than child pornography does.
We wait until we can find a method with benefits commensurate to its costs. A 2016 DoJ report said that since 2002, 10,500 victims have been located and identified by law enforcement. That's 750 per year. There are 74 million children under 18 in the US, so that's 0.001% of children per year. 10,500 people is only 200 people higher than died from "accidental suffocation and strangling in bed" over the same period according to the CDC. The scale of the problem is the same, so should we go ahead and let Amazon send pings to the police if your Alexa thinks your suffocating in bed too?
If in the US Do you like the concept behind 4th amendment? being free from unreasonable searches unless probable cause can be articulated that you have committed a crime?
It is very scary how effect the "think of the children" narrative is to get people to willing give up their liberty
And the way it’s being done (hashes), a collision is highly unlikely. If it does occur it doesn’t mean it’s similar in nature (e.g. innocent picture of own child in bath). The hash isn’t looking at the image content in the sense of “what’s in the picture”, just the bits of the file. So it’s highly, highly unlikely, even if a collision occurs, that the collision would be an image that happens to be another child innocently bathing.
That's not good enough given how our media currently works. Imagine articles published when information about such a check leaks that say "celebrity X's phone checked by police for suspected CSAM." While that is the truth ("suspected") no one cares about that nuance and such a person would get cancelled very quickly, even if there was no evidence of wrongdoing.
Additionally, this isn't just a hash of the file but a perceptual hash on the image content. So e.g. changing a single bit in the image would create a different cryptographic hash, but generally not a different perceptual hash.
Are you sure, given how many iPhone takeover/jailbreak bugs regularly exist, including the recent 0-click iMessage bug? Would you like to dare a hacking group, domestic or foreign, to land a single verboten image on your iPhone?
> And the way it’s being done (hashes), a collision is highly unlikely.
The limited details so far are suggestive that its using perceptual hashes, which are more susceptible to collision-engineering/false-positives than cryptographically-secure hashes.
That's 115792089237316195423570985008687907853269984665640564039457584007913129639936. Probability of collision (two different inputs generating the same hash) is infinitesimal.
Actually, it's very easy to end up with an image that has a similar perceptual hash to an illegal image.
They are not doing MD5 hashing, they're taking perceptual hashes and then using something like the hamming distance or Levenshtein distance to make a fuzzy match with hashes from illegal images.
I've built products using these methods, and it is incredibly easy to make a fuzzy match based on perceptual hashes from two images that have nothing to do with each other.
It's absurdly easy to change the hash of a file. In the case of something like JPEG, you don't even have to change the file itself, you can just change the metadata. Apple could presumably only hash the image itself, but again, all you have to do is make tiny, imperceptible to humans changes to the image and the hash is totally different.
Long story short, this is either nearly pointless and privacy invasive, or it's about to get drastically more invasive to be effective.
Of course, there's no telling what images will be in the database when this inevitably expands beyond CP.
Edit: It makes sense that such a notification system would be impossible to implement. Perhaps, registered offenders should not only have their location placed on a map but their phones have certain privileges disabled or monitored.
The same does not go for random creeps wanting to take close up pictures of a child.
The creep behavior is the problem. The photograph is a symptom of it.
The list ballooned to over a million names, then required an ACLU lawsuit until humans became involved and a review process was initiated.
As long as the majority of Americans dont feel burdened by something, dont expect anything logical to happen.
They are using a library of human verified images to compare the hashes. Probably similar to PhotoDNA[0]. There can be false positives, but AFAIK the algorithms are not trying to identify naked children, but comparing 2 similar image hashes.
I didn't get that from the article, could you point to a source on this?
0: https://www.missingkids.org/theissues/csam#:~:text=Electroni... (ctrl-f "1,400")
So even if not running hashes against known porn, it could still be doing facial recognition. Wonder what kind of implications that would have for people in witness protection or battered spouses and other legitimately "disappeared" people.
“Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.”
Also from the article.
‘Trained’ being important.
And what, in this context, is "your data?" Is a photo on an iPhone Apple's data?
I'd argue that the thing you should do if you get a hit is to delete your data, unless you want to spend enormous amounts of time and money defending yourself from prosecution by people who believe that false positives are vanishingly rare.
I really don't care about the semantics of data ownership. If Apple wants to scan photos you upload to iCloud so they don't run into a scandal years down the road that "iCloud is being used to distribute CP" then that's their call.
And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
No FBI is called. Imagine the financial damage this would do if specialist come over and try to collect evidence and the company is forced to cooperated and work for them, for free ofc. Only the big tech does this. They have whole teams for this and automated system and what not because surely the FBI wont walk into googled data center ever to collect evidence. But all the smaller companies they cant afford to do this and the detection system is actually to protect them form going bankrupt because they accidentally stored some illegal data from a user.
>And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
Yes such laws exist. So they use filter software from third parties. They intentionally offer some generic filter that includes "unwanted content" and doesn't specify what it is when it find something so the companies can wave away the liability as they have no way to know if the image was valuable evidence of a crime or just some LiveLeak gore. They can justify that no human watches it because that's to expensive.
Thank God, I don't use Apple.
(I should probably delete them, since I've basically never used them... although they could be useful for one of those end-of-world Raspberry Pi builds: https://back7.co/home/raspberry-pi-quick-kit-one Edit: the non-bad ones, in case this gets misconstrued)
Edit: I've deleted my archive.
I mean, that is actually illegal. The problem here is the possession of child pornography.
Hoarding porn is kind of useless nowadays, especially without any kind of selection and categorization, but maybe someone does it because it's their hobby (or obsession), who knows.
Also if a lot of people has a bunch of porn images scraped from forums it's quite possible that eventually some of that might become illegal as the laws change here or there around the world. Should they proactively delete everything? Sure, they could. To be safe.
But this kind of safety is absolutely the bad kind on the long term. (Though of course it's possible to advocate for reform while not storing content with uncertain legality.)
There are words like “neural” and “trained” here, but no clear sign that there is actually any kind of image recognition that would risk false positives other than hash collisions.
Thought it was buck naked — but that works too. :-)
It's going to be used in China against enemies of the state.
Then it will be used domestically for the same.
Apple is a horrific company that you should stop using.
>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)
https://www.nbcnews.com/technolog/your-cloud-drive-really-pr...
There was another round of outrage when the Windows 10 TOS extended this to local storage.
It also has the use case of scanning for DRM violations. (ripped mp3 files or movies)
These are still illegal in the US right now right? It's been a major criticism of these laws that they can hurt people accidentally.
It's fuzzy hash, not ""AI"", based. Cloudflare uses it too, and last time i checked the web is still functional.
https://support.cloudflare.com/hc/en-us/articles/36004610611...
How about we turn the tables and have the complainers suggest a solution. Because every single time an approach to targeted child porn takedowns has been suggested, such as datacenter raids which would not affect as many people, someone is screaming about their privacy.
Child abusers evolve and are very happy if law enforcement doesn't.
Fuzzy means that it takes compression and the like into account, because even if just one pixel out of 20 thousand is different, the hash is different too. Fuzzy hash still recognizes it as the same image, so using an algorithm to alter the color etc. won't work.
That's also true for the no-fly list and the Terrorist Screening Database,[1] yet those are full of false positives. And unlike those lists, CSAM databases cannot be independently verified. To do so would require having the original images, which is illegal.
1. https://en.wikipedia.org/wiki/Terrorist_Screening_Database
So if you're charged on the basis of a fuzzy hash matching, you'd subpoena Apple for the photo in your backup that matched, present it to the court (since it doesn't actually matter if it's CP or not to be admissible), and you win the case.
0. https://www.johntfloyd.com/the-difficulty-with-criminal-evid...
The definition of child porn varies around the world. These systems use the US definition. This is not entirely what you might expect. For example, in the USA the courts have decided that cartoons can be child porn even though no actual children are in the picture. Most of the world does not agree with this, meaning an image can be CP in one place but not another. Is Apple going to enforce the US definitions or the ones where the user actually lives?
In the USA, photos an under-age person takes of themselves can also be considered CP.
What counts as a "child" for sexual purposes also varies around the world. Some countries have a lower age of consent than other places. In some parts of the world the age of consent and the age at which a child stops being a child for CP purposes are different, meaning that a teenager can have sex legally but if they take a photo of themselves doing it, they are trafficking in CP.
Finally, what is actually on these image blacklists? Hardly anyone actually knows because of the third rail nature of CP. Tech firms are often delivered image hashes, not even the images themselves, by third party 'charities' of various kinds and tech workers are - for obvious reasons - not normally given access to the actual pixels. Additionally, appeals from users are invariably ignored because people say "legal issues, it's complicated" and so everyone clams up. If FPs occur there is no way to resolve it and the people who see your appeal, if there even is one, won't be willing to actually look at the image to find out what it was.
It should be obvious how much potential for abuse this hands the people who actually manage these CP databases. Literally any image can be made verboten immediately, without any recourse, and basically nobody will ever find out including the people who shut down the affected users.
No. NeuralMatch was “trained” using 200,000 CP images. “Neural“ is likely a reference to the perceptual matching that it uses. It is not a bit for bit match.
Perceptual matching is a technique used for categorizing images based on characteristics and content.
The algorithm will scan your library containing new information and compare it to what it understands as CP.
The specific subject of child abuse is irrelevant in my commentary. It was a commentary on the general category of AI, used all over, for many things, and more and more every day, but nice try.
Edit: You edited out what I was referring to as I was replying.
Being worried about privacy, establishing a precedent for scanning my data against a government database, and the risk of false positives with such an insanely emotionally charged crime is more than mere complaining.
The onus should not be on me to justify why this shouldn't be done. This is something new and it is perfectly fine to argue against it without needing to provide an alternative.
That being said, my solution is to continue to follow the process that law enforcement is currently using.
If you knew how they approach this you wouldn't be satisfied either
Yes. And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system. The only effect this will have is that moral panickers like you will have got everybody's privacy invaded over your moral panic of the day.
This is, and always is, a game of cat and mouse. Law enforcement is always catching up. They are the cryptologists here. They are never ahead, always behind, because they don't know the new protections peddlers are using until they have been in use and later discovered.
No matter what vector you plug, they will use another, and the game continues (sick game). Maybe divide the image into 32 different quadrants and rearrange them, then put them back in the correct order when viewing through a specific image viewer. I'm sure that would bypass whatever detections they've come up in their fuzzy fingerprinting with as the entire image is now different. By the time they catch someone using this, they'll have already moved on to something different, as they always do.
I will never be ok with warrantless searches of my personal property, no matter the reason or justification or subject, and no matter who it is done by (government or private company). And I say that as a survivor of some pretty horrific shit as a kid to the point I fucking tremble with absolute rage when thinking about it 35+ years later. I would be banned from everything for life if I were to honestly state what I would do with these types of people. The movie "Saw" is tame in comparison. I have no compassion or sympathy for these sickos. But when reading world history, I can absolutely see the importance of "innocent until proven guilty" and Blackstone's Ratio "It is better that ten guilty persons escape than that one innocent suffer." Most of human history was the opposite, and it was brutal and full of literal witch hunts. Are we progressing as a species, or regressing in terms of human rights when it comes to technology?
They don't use simple file hashes to match images, but perceptual hashes. That way they can find modified derivatives of a source image. The problem with this approach, though, is that this is ripe for false positives. Two completely unrelated images can have similar hashes.
If they're using fuzzy matching with perceptual hashes, then the space that false positives can exist in for each perceptual hash is huge.
People can argue against this approach for privacy reasons, but I think the false positive argument is a relatively weak one.
There will be many false positives, they will be reviewed by people. When there’s more than a few false positives, you will be investigated by the FBI.
Again, Apple nor any company, have access to the source data, just hashes.
They did add a classifier to iMessage. But it's designed to prevent children seeing any sexually explicit images.[2] There wouldn't be a reason to train it on images of children specifically.
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[2] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...
Perceptual matching is used to sort categories of images. A quick DuckDuckGo will turn up many results. No stretch of the imagination will turn this into a bit for a bit comparison. This is a machine learning algorithm used to categorize images. https://www.ibm.com/blogs/research/2019/10/learning-implicit...
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hashing”
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
“Perceptual hashing is the use of an algorithm that produces a snippet or fingerprint of various forms of multimedia.[1][2] A perceptual hash is a type of locality-sensitive hash, which is analogous if features of the multimedia are similar.”
https://en.wikipedia.org/wiki/Perceptual_hashing
The goal is to verify a black and white copy of an image is identical to a colored original. Search algorithms want a similar thing so they can validate an image contains a blue car. However, a perceptual hashing algorithm must differentiate between different images containing a blue car while matching a photoshopped copy of the same image.
I would hope most privacy conscious people disable iCloud, but that’s another story.
You can setup secure encrypted backups, but the customer losing the key means losing the back so that’s not what consumer focused companies are going to do. In other words any backup service that doesn’t have big warnings that losing your key loses your backup means they can read your data.
"The mud puddle test: You don’t have to dig through Apple’s ToS to determine how they store their encryption keys. There’s a much simpler approach that I call the ‘mud puddle test’"
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
To be clear each image, the image’s NeuralHash, and a visual derivative are uploaded to iPhoto. This allows for the inspection of the NeuralHash algorithm used which I actually prefer.
The phone isn’t downloading the hash database.
Citation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
The perceptual hashing is based on AI techniques. “The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process.”
The difference is AI classification is based on defining something as say a “Cat” and then the AI spits some association with how cat like the image is. This extracts features from an image then compares lists of features to specific images.
From the PDF:
"The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements.
The neural network that generates the descriptor is trained through a self-supervised training scheme. Images are perturbed with transformations that keep them perceptually identical to the original, creating an original/perturbed pair. The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. "
Image classification on the other hand cares about if the image contains say a stop sign or a trash can. That’s useful for self driving cars etc.
Aka classification you might want to match two different bands playing the same song as identical. Where perception hashing would want them to be classified differently.
It is a perceptual Hash of the images characteristics and perceived continent by the algorithm, that’s the “neural” in neuralmatch.
Apple, for yours has been building-in machine learning dedicated chips into their builds so this shouldn’t affect battery life.
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hasing"
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
Which isn't true for Apple. Not too sure about Android. But everything you do on your Apple Devices, including its Hardware, Software and its Services, works more like Apple lend it to you for a price.
And as other mention, only the government and not a private company.
How would that match a hash list of known CP imagery though?
I mean, who's to say that my hash is actually what they think it is...
...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone?
"AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!"
"DEFENDANT: I'm not going to give you my phone's password."
"THE COURT: Bonk: Stay in jail for contempt."
I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
>> I mean, who's to say that my hash is actually what they think it is...
Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1]
"The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate.
And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches)
1. https://en.wikipedia.org/wiki/Universally_unique_identifier#...
Then they could send you the new image and fuck up your life.
If we're talking about random files, sure.
What if the files are intentionally created to match the desired hash? The malicious actor doing this could be a private party intending to disrupt your life, but it could also be law enforcement as a means of gaining access to your device when it would otherwise require a warrant.
This seems a little alarmist to me.
So why would we open up new ones?
This is digital biowarfare material of the worst kind and should be handled as such.
Any researcher downloading this material to anything but lab machines should probably receive a visit from the relevant authorities to make sure they understand the seriousness of it.
That's called a second-preimage attack and is similarly infeasible to achieve against a cryptographically secure hash function.
There are good arguments to be made for why this is a bad idea, but hash collisions aren't one of them.
This is a function that is created to be robust against modifications to the image.
A perfectly robust function would - in my opinion - be one that could detect even a recreation of the original setup.
I don't know where on this scale the function law enforcement use is, but if it is any good I struggle to see how it cannot detect similar legal imagesm
Nope, federal law states they can hold you for 18 months. So long as they can claim to already know what's on the device (which they will, because Apple pinged them about it), they can hold you for 18 months.
Definitely a pretty thorny situation.
https://www.aclu.org/other/constitution-100-mile-border-zone
- Add the hash of a non-pornographic image to the database
- Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it.
- The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you.
But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
I have no illicit images, but false positives are always going to be a problem and even at sub-1% rates if you're scanning literally every image stored on an iOS devices that could still be thousands of wrong matches. If I lose the false-positive lottery, am I going to have the police calling and have my mugshot in the evening news for "CP on their device" in particular in my state's months long backlog for digital forensics. Law enforcement aren't exactly known for their understanding of technical evidence like hash-matching.
I'm very sympathetic/supportive of anti-CP initiatives in general, but this still seems problematic.
PS - And before someone replies with "they aren't contacting the police," don't forget the "yet." Once this is in place they'll immediately get pressured by politicians and well-intentioned people alike to flag any positive detections to authorities who will overreact (as law enforcement has a history of doing, particularly around this topic).
> The automated system would proactively alert a team of human reviewers if it believes illegal imagery is detected, who would then contact law enforcement if the material can be verified. The scheme will initially roll out only in the US.
This is dystopian and needs to be opposed, not applauded.
As others mentioned, privacy erosion only goes one way. This is a frontier being breached, i.e. searching your physical device without a warrant, and as usual in the name of CP (the alternative typical pretext being terrorism). It won't take very long before this gets extended to any illegal activity. Gay dating was an illegal activity until not so long ago. This sort of chinese-style monitoring is not desirable in any way, and I find it appalling that Apple seems to want to be the sponsor.
Apple’s system is less invasive in that the screening is done on the phone, and “only if there is a match is notification sent back to those searching”, said Alan Woodward, a computer security professor at the University of Surrey. “This decentralised approach is about the best approach you could adopt if you do go down this route.”
Don’t need to MITM when you put an M at an end.
If they get a subpoena and then maybe send some kind of file list/checksums they've been required to hand over to some 3rd party / government? That's possible, but an entirely different situation.
Furthermore in principle there's no reason this has to be, or will be, just CP. First it'll expand to blatant terroristic images, say ISIS beheading type videos. Then it will progress to subversive material: COVID "misinformation", "domestic terrorism" (read: anyone the feds don't like), the list goes on. This tool and its successors will be used to completely abolish the concept of device privacy and will be used to ruin the lives of anyone who sufficiently pisses off the feds. And that's just in America, god knows what they're going to do with this system in China.
2. This article only provides evidence to say Apple will scan photos you upload into iCloud. Scanning local photos is your speculation.
3. The operation of NCMEC and the blacklist is defined by the Federal government. American (citizens) have some mechanism to oppose this design. Apple could perhaps lobby against it, but their lobbying operations are famously minimal.
4. In principle, Apple could do anything. That doesn’t really inform what they’re likely to do. Just like in principle, anyone could slip a malicious content scanning patch into the Linux kernel (and this already has a POC!).
Ok so they’re planning to eventually roll out this spyware worldwide. I do not know how Americans can think China is bad in this while allowing this to happen.
Also, while there's a lot to be said about court systems, public defenders etc in US I wouldn't compare the two.
Also China doesn't count Uyghurs as incarcerated despite the fact that the PRC is committing genocide against them.[1] The PRC imprisons almost 2 million of Uyghurs a year. Of those, around a quarter million die. Oh and there's the forced sterilizations, destruction of mosques, brainwashing, banning of muslim-sounding names[2], forced labor, organ harvesting, and mandatory quartering of government agents.[3]
1. https://en.wikipedia.org/wiki/Uyghur_genocide
2. https://en.wikipedia.org/wiki/Naming_laws_in_China
3. https://en.wikipedia.org/wiki/Civil_Servant-Family_Pair_Up
China doesn’t count Uyghur extremists as incarcerated for the same reason the US doesn’t count their brethren it kills en masse in Middle East as death sentences. It’s just a different thing.
As for the rest of the list - most of it is just US state propaganda calling resocialization programs “brainwashing”, or leveling commercial buildings built without building permit “destruction of churches”. Some of that works exactly the same in US (forced labor), or is just entirely made up (“quarter million die”).
Now, some of the stuff is a real problem of course - death sentences and organ harvesting - but throwing it together with the usual US propaganda doesn’t help the cause.
That's not true. If you released everyone convicted of a drug crime (including manufacturing, selling, or distributing), you'd decrease the US prison population by 20%.[1] I'm all in favor of ending the drug war, but that won't do much to the incarceration rate. Also, the drugs that are illegal in the US are just as illegal in other developed countries (if not moreso). To use one example: in the UK, you can get up to 7 years in prison for simply possessing LSD or shrooms, and life in prison for for supplying or producing them.[2] No european country has legalized recreational marijuana[3], while the state I live in legalized it a decade ago. Now look at China's drug laws. While in China I've been solicited for all kinds of illegal goods and services, but never drugs. Stolen or fake merchandise? Sure. Prostitution? Of course. But drugs? Not a chance. The reason for this is that China executes people for drug possession. They randomly test foreigners entering the country and prosecute them even if they have no drugs on them.[4] The government raids nightclubs and residences, then forces everyone to take drug tests and prosecutes them if they test positive.
But please go on about how oppressive the US is.
I've seen you comment on China several times before and it's always in favor of the PRC. This confuses the hell out of me. Consider this: We could not have this discussion in China. To contradict the official narrative there is to risk your career and your freedom. Yes, the US and other countries have big problems, and it's good to try and fix them. But the level of surveillance and oppression in China is unlike anything in the west.
1. https://www.prisonpolicy.org/reports/pie2020.html#firstmyth
2. https://www.gov.uk/penalties-drug-possession-dealing
3. https://commons.wikimedia.org/wiki/File:Map-of-world-cannabi...
4. https://www.gov.uk/foreign-travel-advice/china/local-laws-an...
Regarding other countries - what would happen (and happened) to countries that refused to join War on Drugs?
Regarding possession - in UK essentially nobody cares, unless you’re distributing industrial quantities and/or are really obnoxious about it, eg walk into Constabulary smoking a joint, or by putting up advertisements. There are regular news about eg police finding a suspect car parked for six hours with engine on, filled with smoke (actual example I remembered because it’s funny) and they don’t result in charges. I suppose the law could be used if you also committed other offenses.
As for not being able to talk politics in China: where all the policy and law changes come from then? And if you can freely talk in the US, why so many people have many throwaway accounts here on HN?
Anyway: yes, China is oppressive in a number of ways. It’s about one generation behind the US, except for racism, where it’s ahead. US itself is perhaps a generation behind Western Europe. But everyone knows what’s bad about China, and everyone knows what’s good in the US. Anti-Chinese propaganda is everywhere, and I find it quite annoying. Particularly so because it detracts attention from real problems - if American propaganda is lying about genocides and missile silos, why assume other parts are true? Secondary: all the bad stuff in China stays in China. They don’t routinely exterminate hundreds of thousands people in other countries for financial or racial reasons. They don’t intentionally ruin other nations’ economy. The US does. And I feel like this perspective is entirely lost among western population.
Or someone that doesn't like you can MMS/iMessage/e-mail you images that match PhotoDNA image hashes. Maybe even in a way where the images don't appear to be anything alarming.
Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them".
But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
(1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes
(2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
> although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
Only in the sense that "we don't know of a way to generate a new sample that has the same hash, yet". Unless we can prove P!=NP, we can't even prove that there exist hash functions that are actually infeasible to generate a sample with the same hash - so far we are no where close to knowing whether P!=NP or P=NP.
Even if we assume P!=NP, that doesn't mean that it is infeasible to generate collisions for our current cryptographic hash functions. They're not backed by some mathematical proof of correctness assuming <assumptions>, just the fact that so far no one has publicly figured out a way to break them, and the people who spend time trying think that their design is one that is unlikely to be broken in the near future.
We have managed to find collisions against hash functions that we previously considered cryptographically secure, that were designed with the same sort of standard in mind: E.g. md5 (https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities) and sha1 (https://en.wikipedia.org/wiki/SHA-1#Birthday-Near-Collision_...).
There's no great reason to think that private groups could not have an attack against current hash functions [1], and there's definitely no reason to think that they won't find one in the future.
[1] There sort of is for SHA2, and that's that bitcoin has effectively created a giant bounty for breaking it.
Linux truly is the last bastion of sanity. I am so glad that linux desktops took off and work as well as they do. Last thing I should be worrying about is if an employee at Microsoft or Apple decides their "automated system with no errors" decides that a hash collision or a picture of a baby is close enough to actual abuse imagery. There are plenty of people that believe even a picture of a naked baby in a bathtub is abuse imagery. Anyone remember the case of Walmart? They took their kids away "pending investigation".
https://jonathanturley.org/2009/09/18/arizona-couple-sues-wa...
I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
All iCloud Data in China are already within their control. The whole DataCenter are done with "partnership" agreement.
HK user have their Data in US but I believe there could be a temporary server in HK DC before it is sent to US.
I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.
Apple has to comply with the law, however.
There is a federal law requiring electronic service providers to notify if they spot such material. Apple may be subject to that. See the “Federal CSAM Law” section here:
https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-b...
Presumably, they are doing this voluntarily now to stave off future legislation around this that might require more invasive scanning.
1. 18 USC 2258C(c)
A big, rich company like Apple will have some leverage, but if the state is determined that won’t matter.
Our controls on the state will need to be outside of private companies.
https://quotepark.com/quotes/1911588-immortal-technique-the-...
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system
Why is there any need for Apple to install software on the iPhone if they are isolating the algorithm to run only on cloud storage, not local images? Not a programmer, so maybe a simple explanation.
Also, if this is isolated to iCloud, won't criminals just start using a different cloud backup provider?
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"?
If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives.
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities
Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.
https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...
That would be Steve Job's Apple. Tim Cook's Apple will think they are doing it for the greater good.
The road to hell is paved with good intentions
- Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household
- Fund undercover police work and informants, which is how most large busts already happen now. Ideally highly independent from existing courts and law enforcement to deal with Epstein- or Dutroux-type situations. The worst offenders are not swapping child porn on Facebook or Dropbox (or now iMessage and Apple Photos)
- Enforce obscenity laws against incest and other extreme forms of pornography
- Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly
Since these aren't under consideration, we can tell this isn't about reducing child abuse. It's about setting up a permanent and ever-expanding digital system for political control.
How would you go about enforcing this, and how does that differ from existing proposals about child porn?
Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly
All the worries about people being framed or wrongly convicted apply equally well to that proposal.
This topic already made it to the current frontpage.
Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
Apple has root on every iPhone. They can do whatever they want by pushing whatever code they like to any iphone.
This doesn't really make any sense. So it's not a Neural Engine thing but merely hash matching, and it also only applies to iCloud Backup? That is already non-e2e encrypted, so Apple can already access those files whenever it wants.
I just ordered a Iphone 12 Mini to replace an Iphone SE, but now I’m going to return it. No amount of cool tech in a product is worth enabling this slippery slope against human privacy rights.
I mean, if you get a really good nude from someone you're going to want to keep it. But for how long? Is there now a social rule of how long to keep nudes for? Is 21 too old, and you should delete them? What if you're still with the person?
Personally, I'm glad I don't have to figure it out, but teens sending each-other nudes just has all sorts of confusing ethical quandries to it.
Anyone that trusts the feds to do the right thing here is incredibly naive. They get to operate a black-box database; if your phone has any photos that match an entry in the db you will be reported to the authorities. I can't believe on HN of all places there's not 10x more of an outcry.
I believe it's not "an entry in the db" but a neural classifier. The article calls it "neuralMatch". So it's much more general than a limited number of known images.
This is sort of the premise to why I've de-Googled/de-Cloud myself because I didn't want my little girls pain and suffering to train some computer's image detection AI.
How would this work for a parent of a sick child who sends their doctor, S/O, possibly graphic in nature and personal photos of their own child? I was in close contact with my daughter's doctor, and would send him all sorts, I can only imagine the same is true for most parents of chronic children.
As many others have noted - the potential for misuse is rife. Also, realize that Apple has a single set of infrastructure for their phones and macs, especially now that M1 exists. I've read accounts in the person where disaffected people "planted" porn on other peoples system, then called the police.
Imagine a web server with a page and a hash map matching image, with a frame-size of 1px. It's now in your cache.
Also, it's almost inevitable that this is a slippery slope. Today is may be pre-hashed images. Tommorow it may be AI detection. The day after it may be vaccine falsehoods. The day after it may be other bad government speach.
This is needed. This is a major major problem. But I hope someone is thinking of how to mitigate the potential for abuse.
But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
Just like the Patriot Act and the no fly list, it's only a very short matter of time until this is expanded to Android and more importantly, non-CP content. The only difference is its the other party doing it this time, in both cases its "to catch bad guys" and "for your own good".
When I look at the technical details, it seems to me to be a reasonable compromise. It allows Apple and government to do something about the worst offenders, whereas it has no impact on anyone else.
Two technical reasons for this:
- The "neuralMatch" algorithm suggests some sort of CV, whereas the article talks about matching against a hash of know images. My guess is that the actual technology is something like Microsoft's PhotoDNA (https://news.microsoft.com/on-the-issues/2018/09/12/how-phot...). Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.
- As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow.
- It sounds like it doesn't flag a single image, but requires multiple hash hits.
So, if you want to feel better about this, understand that it is a system that will flag people who are downloading storing known child exploitation images on their devices and naively backing those up to iCloud.
This is the sort of privacy compromise that works in practice. Serious offenders are either caught or diverted to other channels. Minor offenders are probably not caught. The risk to non-offenders is zero or close to it.
Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
If we had a way to guarantee that it would stay that way, it'd be less concerning. As it stands, clearly there are plans to scan on the device itself, otherwise there would be no need to roll out software on iPhones at all.
> Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.
Agreed, but the issue is the government can stick whatever photos in the DB they want. And we have no way to verify what's in there, since they're not gonna release thousands of CP images for us to audit them.
> Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
It changes plenty. Currently Apple more or less promises not to spy on you, and so they have to actually be lying to do so. Is lying an option for them? Sure, but that's different than them just announcing in advance "hey we're gonna start scanning every photo you have and crossreferencing against a somewhat-transformation-invariant secret government database". I'd prefer that they have to lie to me to spy on me, since that at least has the potential for backlash if discovered. If they get this feature rolled out we're all fucked.
In 2017 a 369% increase in CSAM in terms of reports happened. The only explanation is that platforms now facilitate the sharing and a growing problem continues to grow.
In raw percentages the data clear: too many children are being abused and the numbers are growing.
https://storage.googleapis.com/pub-tools-public-publication-...
This idea of perpetually renting services than monitoring the use is something that should be criticized.
ie, have private company give themselves the right to go have a look through your belonging and in your home because you might be a criminal.
It is now well known that most citizens of most countries are criminals, so no one deserve privacy anymore...
But sadly, it looks like that intelligence and common sense has generally decreased in the population.
Btw, the appleTV and iphone, that are always listening to your conversations in your home to detect some bad keywords and then denounce you to the police is not yet ready? Nazi collaborators, from their tombs, are still waiting for this nice feature!
Anyway, iphone owners, let me tell you that you did a good job by stupidly giving your money and power to Apple!
The 1st and 4th amendments tell the fed to go pound sand, Apple should follow suit.
It also means that you can't have any trust about your data safety even if you never put it in the cloud.
It seems not, so we have a choice between this and Google's always-track-you setup.
I'm sure no one here has any illegal pictures, but the precedent is terrifying. What about copyrighted images or downloaded videos without DRM ?
I put myself through school working at a one-hour photo shop in a drug store. It was my job to feed the machine, keep the chemicals balanced, paper loaded and perform white-balancing and cropping to make sure the end result matched customers' expectations. It was a relatively small town, and I lived in the same neighborhood that I worked in - so I basically saw 80% of my neighbor's photos during the development, cropping, and packing process. I saw hundreds of baby's in bath tubs, and bare a* toddlers on the run, in the mud, etc. Also saw a lot more bedroom photos than you (or at least 17yo me) might expect.
In the three years of work, I only had one roll of film for which I called the police. It was obvious child abuse. I stopped the machine, called the police and they collected the film directly into evidence and made the arrest when the person came to pick it up. The guy in question thought that the 1hour process was "all automated" and he was pissed that somebody was looking at all his photos.
Not sure that has anything to do with the iPhone AI scanning - but it might be an anecdote of interest to those that don't remember a time when somebody looked at all your photos as part of the creation process.
Where is my right to refuse to dedicate my devices cpu, battery time and network bandwidth for this activity?
Did I agree to this already?
(Sorry for asking a question! I'll try to simply trust Apple at all times from now on...)
I also would like to know if I already agreed to this and where can I read the agreement that I signed?
CSAM also means Cyber Security Asset Management.
IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.
You can request a review of Apple's actions, however we are receiving higher than normal requests at this time. This means we may be unable to review your account.
Sorry for any inconvenience.
You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are.
Edit:
Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this? ____________
Second Edit:
Wanted to reply to another comment but being rate limited:
As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities.
The number of Muslims in America:
A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population.
Lol. Jesus Christ, you do the math.
No. That's the beauty of the third-party doctrine. Your rights disappear in a puff of twisted logic because you agreed to the fine print.