Show HN: 59a34eabe31910abfb06f308 – NeuralHash Collision Demo
thishashcollisionisnotporn.com
thishashcollisionisnotporn.com
Thank you for caring enough to put this together and publish it.
We should cool down a little and wait for the real system, before expecting anything from them. Let’s see then how bad it is, now we are just victims of speculation.
However, this is something which have been planned for years. Bad PR won't undo this, and stepping back is not happening overnight.
There is so much misinformation in this subject, because it is really hard to see the bigger picture. We are in this situation, because of the leaked information. Was it intentional for political reasons, who knows.
Apple is definitely going to announce some E2EE features very soon, such as backups in the next event. It is very hard to invent better system, than they have released now, to enable E2EE and apply CSAM scanning.
When you step into the Apple system (which is quite closed, and nobody really can make extensive review for it), you give all your trust for Apple. Whether this scanning happens on your phone, or in the cloud, it does not really matter, as long as the end result is the same (same files are scanned).
How about the abuse of the system? Prior on-device scanning, government could ask Apple to add feature for telling, if users have Winnie the Pooh images on their phone. (Could you spy for us?) After adding the on-device scanning, now they ask Apple to add this hash dataset to find Winnie the Pooh images. (Could you spy for us?) The question is still the same. Does some technical change affect for the managers and the chair?
How about forging CSAM images to make FBI to show up in the neighbors door? Not going to happen easily. The only problem here is to brute force their human review process. In that case, it is further double checked by NCMEC. But still, at first we need some hash which exist in their database, and secondly that image should be uploaded to iCloud. And not only once, 30 times with different hash.
That's what Apple wants you to think, and they're very successful at marketing, but it's not really true. The jailbreak scene is not what it once was, but there are innards to Apple devices and services. Apple engineers put their pants on one leg at a time, the same as everyone else.
[1] https://www.theguardian.com/technology/2019/jul/26/apple-con...
It seems to be getting a lot of attention because it's an easy target, but it doesn't seem to be a very meaningful one given the system they say they use.
It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)
Apple isn't exactly known for software transparency or allowing software that monitors iOS internals.
Many reasons this is bad.
Obviously the "Slippery slope" argument around it scanning more in future and being abused by the state.
There is the possibility that its exploited/bug that sends data when it shouldn't or falsely reports you (not a collision). Eg. a bug that reports all photos as bad instead of listening to results of scan.
Obviously the possibility of collision sucks. But thats somewhat constant across implementation options.
It wastes your CPU resources and network resources to do something that only benefits apple.
It's a bad precedent because other companies will likely do this now, repeated the above arguments with increasingly less trustworthy companies.
It seems that this system might be extremly resistant for collision to prevent them ending up to human review.
https://news.ycombinator.com/item?id=28305946
> It wastes your CPU resources and network resources to do something that only benefits apple
Okay, how practical is this problem? iCloud already scans your files to know whether they are need to be synced. And this scan applies lesser times, only once before upload per file.
Encryption does not mess with the filesize, so we are talking about few bytes of extra metadata per image here. Regular image with raw format is tens of megabytes.
> Encryption does not mess with the filesize
No but giant bloom filters take storage.
Building these types of systems is antithetical to a free society.
Can you predict the US' political climate in twenty years' time? No? Then don't build this.
And how do you propose that all of these evil regimes are going to get their images into the NCMEC database? The hash DB will only include photos that are in NCMEC and a second countries’ CSAM database.
And it will be trivial to verify that the hash DB is consistent across different countries.
All they have to do is say: in order to sell iPhones in this country, you must do what we want. Apple will capitulate.
I am russia.
I take actual child porn from my vast kompromat databases (perhaps sent to me helpfully by facebook), or have my agents make some more themselves. They have many talents.
I use adversarial modification to make the child porn match many images of gay pornography popular with the peoples of my country. I add these modified images to my child porn to my databases and also ship it off to the relevant agencies in other countries. It's obviously child porn, so of course they add it.
Apple staff forwards me matches, the images look pornographic and they hit the database. Failure to report child porn you discovered is a felony, so they will error towards reporting.
If for some reason apple doesn't forward on enough of the matches, I hack their severs, kidnap their staff, or simply order them to provide the data they are already collecting (on penalty of not being able to sell in russia anymore). I can continue to use the pretext of searching for child porn to do all this with a smile.
I think this is all obvious enough, and I'm sure the people who work in this business are smarter than we are, have capabilities we can't imagine, and can come up with even better attacks.
[0] It's federally illegal for anyone to have CP on their system, or to view it.
[1] So the workaround is displaying a distorted low-res version instead https://www.apple.com/child-safety/pdf/Security_Threat_Model...
[2] You can replace CP with anything your oppressive regime is trying to snuff out.
This is not correct. Having a few will put those photos through a second, blind hashing algorithm. It wouldn't be flagged for human review unless the images in question passed the second stage and, even then, you wouldn't have any of the information for those hashes to match.
Having a few will have the flagged photos sent to their servers, now liable to be obtained by external forces. Then processed by a second visual hashing algorithm, liable to the same type of collision attacks, using obfuscation as its promise of security.
> Once Apple’s iCloud Photos servers decrypt a set of positive match vouchers for an account that exceeded the match threshold, the visual derivatives of the positively matching images are referred for review by Apple. First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.
So to recap, this second blind hash you're talking about is run _after_ the server has already received and decrypted the visual derivatives (what else would it be hashing?).
Perceptual hashing is highly susceptible to collisions. I'm not sure what you're talking about with "hiding a different thumbnail through resizing". All you need is nude-looking colors in your photo to let the derivative look like credible CSAM. No resizing tricks necessary.
The downside it that this makes the human review process tricky. Because there is, understandably, no actual CSAM image to compare the visual derivative with. Images that look similar enough (contain nudity or a child in a swimsuit) may get past the human review process.
Having a image sent to LEO is effectively being accused of a crime.
What prevents a judge from approving a search warrant on the basis of an image that matched a hash? The answer is, of course, “nothing”. What guarantee do we have that these matches aren’t used as a signal to some other automated extra-judicial search? Again, none.
None of this is fantasy. Take a look at the no-fly list and and no-knock warrants and ask yourself if you really want an automated system making accusations to LEO.
e: clarified that images are sent to LEO, not just hashes.
"Those targets will not meet themselves"
"What if that person has CSAM but tried to obfuscate it. Would you forgive yourself if you let it go and it turned out you were wrong?"
There's a fair point in there about public defenders not meeting that bar, at least in terms of the quality of their representation given the amount of work they have, though.
Lawyers aren’t always doing their maximum. It doesn’t matter that they get paid, you also have to be among their top important customers, just like aircon repairmen.
Also, the picture is not CP but by that time, the phone is already entirely searched.
Law enforcement has no incentive to exonerate you, but they have tremendous incentive not to waste everyone’s time, including their own, on fake cases with little chance of conviction.
Sadly, in the US prosecutors regularly mislead the court because our adversarial system of law is seen by many to obligate them to zealously prosecute their case up to the boundary of the law. (That the prosecutor should conceal evidence, mislead, exaggerate, right up to the point where any further and their actions would be an unambiguous crime)
As a result, there have been child porn prosecutions over totally legal material rescued only by the actress in question showing up in the court. ( https://www.crimeandfederalism.com/page/61/ )
At the end of the day, Apple's decision to reprogram their customer's private property to snoop on their users private files and report on them creates a risk for those users. The risk may arguably be small, at least today. But it is a risk none the less.
We don't need trillion dollar corporations snooping on users phones like masked vigilante crime fighters. Batman is fiction, and in even the Dark Knight when he spied on everyone's phones it was clearly portrayed as an erosion of his moral character out of desperation.
People are routinely arrested over the results of field drug tests despite their high sensitivity and high probability of false positives.
I dunno man, I’m not to worried about this chain of events.
When photos are uploaded to places like Google images they are scanned for this material.
Apple is currently only processing images that are being uploaded to iCloud. So, these are photos people intended to upload to them. This means it's not unwanted upload of photos to Apple servers. That is happening anyway.
Note, I think scanning should happen on Apple servers and not the phones. This is an invasion onto the devices in my book.
I wonder if this is a way to scale the system and reduce their servers and power use.
Even Dropbox is using quite similar scanning to identify how to sync all of your files to the cloud, by not re-uploading the whole file at specific intervals.
This system is genius in terms of maintaining the user privacy and locking Apple out of your files. That is something that most refuse to see. They have invested a lot.
If the objective was maintaining the user's privacy they could use industry standard end to end encryption.
If the objective was to offer somewhat more privacy than nothing while still acting as a vigilante crime fighter, they could make the user's software use a cleartext hash database and report the user when there are matches, and otherwise use industry standard end to end cryptography.
Only when you add the requirement that the vigilante crime fighting must occur behind a mask-- that it must be cryptographiclally impossible to hold apple and its sources to account for the content of the database do you actually arrive at the need for substantial new investment.
Yes, anyone can implement surveillance. Apple's innovation is kleptography. A surveillance infrastructure (somewhat) successfully disguised as privacy protecting cryptography.
It is impossible to predict the future perfectly, but what if the most valuable company in the world applies full E2EE services? Governments control companies with regulations, and even before this, there is a lot of talk about compromising E2EE with government access. In Germany, they just passed the law in June to do exactly this.
> compromise privacy while simultaneously shielding themselves and their data sources from accountability at the expense of even more privacy and risk of abuse.
It is true that they shield themselves at the same time, but as the alternative is not to encrypt at all, this is not compromise of privacy. It is a step towards more "E2EE like" data, and does not worse the situation of consumer.
> If the objective was to offer somewhat more privacy than nothing while still acting as a vigilante crime fighter, they could make the user's software use a cleartext hash database and report the user when there are matches
It is impossible to verify the validity of the report in such a scenario, and they have no legal base to act. This is the scenario which would bring the FBI behind your door with adversarial hash collisions, because they can't validate anything.
Sorry I was unclear, I mean in that case it would send them a cleartext copy of the file, and if it doesn't match (otherwise) just do the standard end to end thing.
It would still violate the user's privacy, but it's a simple, cheap to implement, and relatively transparent system (e.g. we could test the database against the internet to discover if they're misreporting popular images) which would continue to allow Apple to cosplay as Batman.
I don't think it's good. I believe users devices have an unambiguous moral obligation to act in the user's best interest to the extent provided by the law unless directed otherwise by the user. I believe we should consider enshrining the moral obligation in the law. Unfortunately, unethical actors have moved the overton window so far over that rather than getting a device agency law we'll have to fight laws mandating backdoored encryption like Apple's.
I bring up the alternative to point it out that the fancy crypto in this new system isn't there to protect the user, as the parent poster was suggesting. The fancy crypto is a mask that helps shield Apple from accountability for their vigilante actions. Even if you take for granted that Apple gets to play crime fighter there are simpler and more transparent alternatives.
Please spend a few bucks on supporting them.
A bit of a background on why apple did this (this was flagged, but I don't know why): https://news.ycombinator.com/item?id=28259622
Others, needless to say, have not had enough yet.
For a similar example, Apple's Airtags have a number of protections against using them for nefarious purposes. Tile, Samsung SmartTag, and similar devices don't, but Tile users are about 2 orders of magnitude less common than Apple users. You could get pinpoint tracking of a person just about anywhere by dropping an Airtag in their bag, but you'd be lucky to pick up one or two pings if you dropped a Tile.
I think the NeuralHash client scanning approach is overly invasive, but at the same time I think it's a good thing that Apple has someone who's thinking about bad things that can happen while people are using their products.
...as if having a trillion dollar corporation playing batman and going on a vigilante crusade to scan your private files is a situation we should already be comfortable with.
Heck they might even produce some ideas how to improve new surveillance tech for free!
Pure brilliance.
I know this is a controversial take (in HN circles), but I no longer believe this will happen. This kind of tech has existed for a while, and it simply hasn't happened that it's been mis-applied. I now think that this technology has proved to be an overall net good.
Maybe someone sends you some unsuspicious pictures and you get a visit by the FBI, or some apple (outsourced) employee reviewing them. maybe it happens with some photo you take, who knows.
This is not a slippery slope, this is what can happen with the process as defined by Apple.
[0]: https://huggingface.co/spaces/akhaliq/AppleNeuralHash2ONNX [1]: https://gradio.dev
So who controls the database?
https://www.theverge.com/2021/8/13/22623859/apple-icloud-pho...
To make an image match a specific hash, you pass the you want to modify image through neuralhash and compute a difference with your target hash, then ask your netural network library to use reverse mode automatic differentiation to give you the gradients for each of the outputs with respect to the input pixels.
Update the input pixels.
To make the collisions that look good, you need to either augment the objective to include some 'good looking' metric, or condition your gradient descent to prefer good looking results.
In my examples ( https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... ) I used a gaussian blur to highpass the error between my starting image and the current work in progress, and fed that back in, and adapted the feedback to be as great as possible while still keeping it colliding.
The bad looking examples that people initially posted just did nothing to try to make the result look good.
I'm thrilled to see that someone other than me has now also worked out getting fairly reasonable looking examples.
Not for the general public, it's also in encrypted form on the device
I can tell you how they will make it auditable without revealing hashes to all of their users: by only showing them to experts, just like you don't publish the entire databreach credentials in cleartext but hand them to someone experienced who acts as a middleman for public transparency
If not, what is going to convince them to stop at this point?
I believe that they should scan user data in some capacity, because this is about data that causes harm to children.
However, I believe that they should not run the scan on the device, because that carries significant drawbacks for personal privacy.
This is a carefully chosen stated reason for Apple to add this functionality to avoid user pushback. The actual uses will vary from country to country, but on the whole will not result in the protection of anyone other than the government or ruling party.
In countries with less civil freedom this will be exploited and used as a tool of oppression against minority political groups.
The on-device scanning capabilities will be expanded in the future, but it can be stopped if Apple care about their users.
Right now it seems they don't care.
Forgot to mention that I'll not be recommending or purchasing any Apple devices until they halt this scanning idea.
Apple cares about selling devices, maximizing revenue. Users are just something required to authorize transactions.
If Apple has to kowtow to the most oppressive regimes in the world to keep doing the above, you bet they will and probably already are. And since it is in the interest of both Apple + Gov't to do shady stuff in the dark, there is a good chance they can hide it the public indefinitely.
I dont care if it will save billions of lives, your devices should not betray you.
There are things you can use technology for to help combat CSAM, but this is certainly not an acceptable way to go about it. Creating tools to help report abuse and illegal imagery in their software, for example, could be useful. This is not.
What if the police come to Apple with a warrant? Should Apple have the responsibility to reveal the user's content to authorities?
Part of the motivation for this type of thing is that E2EE effectively makes the old system of warrants obsolete and hides this data permanently from authorities. Lots of people see that as a huge win. Lots of people see that as a huge loss.
If they lack the capability to do so then they don't need to worry about it.
> Part of the motivation for this type of thing is that E2EE effectively makes the old system of warrants obsolete and hides this data permanently from authorities. Lots of people see that as a huge win. Lots of people see that as a huge loss.
The speculation that this enables E2EE is just that; speculation. It's also not a strong argument because Apple could implement E2EE without snooping people's photographs and they'd be in the same position of being unable to assist law enforcement.
If you have a system in place that intercepts all files before they're encrypted and uploaded to the cloud, then E2EE doesn't serve its purpose of ensuring privacy of people's data. It completely defeats the point.
"They don't need to worry about it" may be enough for you, but it is not a satisfactory moral answer for a lot of people. It shouldn't be tough to understand that some people don't want to actively enable illegal activity.
>If you have a system in place that intercepts all files before they're encrypted and uploaded to the cloud, then E2EE doesn't serve its purpose of ensuring privacy of people's data. It completely defeats the point.
Which is the reason why Apple proposed running this on the device so that those files don't leave the device if they aren't flagged.
You're not "actively" enabling illegal activity. The consequence of protecting people's rights is that illegal and immoral activity can occur under those same protections. It shouldn't be tough to understand that people don't want their rights trampled on to enable surveillance.
> Which is the reason why Apple proposed running this on the device so that those files don't leave the device if they aren't flagged.
How does this solve the issue, exactly? The fact that it's on-device doesn't change the equation here.
I don't want to get into a semantic debate about the word "actively". Suffice it to say that some people don't want to work on a product or support a company that they know is enabling criminal behavior while also refusing to do anything to curtail that illegal behavior.
>It shouldn't be tough to understand that people don't want their rights trampled on to enable surveillance.
Apple recognizes this. Once again, this is why they are doing this on device and not in the cloud like other providers. They are trying to do the scanning in a way that is invades privacy the least.
>How does this solve the issue, exactly? The fact that it's on-device doesn't change the equation here.
It changes the equation because it allows for everything leaving the device to be E2EE. One example is that this would stop anyone at Apple from snooping on your files. If the scanning was done on the server, Apple would need decrypted versions of the files which could be viewed by Apple employees.
I already gave one example of something they could do to help curtail illegal behavior without requiring an invasion of privacy. Some people also don't want to support a company that they know is enabling surveillance, which is what Apple is doing here.
> They are trying to do the scanning in a way that is invades privacy the least.
A violation of your right to privacy is a violation no matter how small you might think it is. They don't need to invade our privacy at all.
> It changes the equation because it allows for everything leaving the device to be E2EE. One example is that this would stop anyone at Apple from snooping on your files. If the scanning was done on the server, Apple would need decrypted versions of the files which could be viewed by Apple employees.
Everything leaving the device isn't E2EE and this doesn't "allow" for everything leaving the device to be E2EE. Stop pushing this false dichotomy. Things can be E2EE without any invasion of privacy whatsoever and there are other options to help combat CSAM that aren't violations of a user's privacy. Stop pretending like this is the only option when it's clearly not.
You said they should add a reporting mechanism, that won't stop anything because if a person already has knowledge that this content is being shared, they could just report it to the authorities. There is no reason to report it to Apple. Like you said, "Apple's role is not the police".
>Some people also don't want to support a company that they know is enabling surveillance, which is what Apple is doing here.
I acknowledged that in my first comment. Some people feel one way. Some people feel another. Apple is acknowledging both sides of this while you are only acknowledging one.
>A violation of your right to privacy is a violation no matter how small you might think it is. They don't need to invade our privacy at all.
This type of black and white ideology can't function in the real world. All rights have limits and the general rule is that your rights end when they start infringing on my rights. Your right to privacy does not supersede my right to not be a victim of a crime.
>Everything leaving the device isn't E2EE and this doesn't "allow" for everything leaving the device to be E2EE. Stop pushing this false dichotomy.
Everything I have seen from people in the know has suggested that this is a step on the path to Apple enabling E2EE encryption for iCloud.
>Things can be E2EE without any invasion of privacy whatsoever and there are other options to help combat CSAM that aren't violations of a user's privacy. Stop pretending like this is the only option when it's clearly not.
Do you have an actual suggestion that you think will cut down on CSAM beyond providing a redundant reporting system?
Recognition isn't something people would always be able to immediately do with CSAM imagery so Apple, which has already created a tool to recognize it, can assist with that. It can also create tools to reduce the overhead of reporting. If you make it easier to report, more people will do it. You're assuming that anyone that ever encounters CSAM would go out of their way to report it, which simply isn't true.
> This type of black and white ideology can't function in the real world. All rights have limits and the general rule is that your rights end when they start infringing on my rights. Your right to privacy does not supersede my right to not be a victim of a crime.
My right to privacy is not an infringement of your rights so this argument has no bearing in reality. Law enforcement requires probable cause to get permission to surveil the population and there's been countless cases thrown out because of this violation, which, as it turns out, was even tested in the context of technology interfacing with NCMEC.
> Everything I have seen from people in the know has suggested that this is a step on the path to Apple enabling E2EE encryption for iCloud.
Those people are also pushing the same false dichotomy that you are. There is no technical reason that this is required to enable E2EE for iCloud, it's purely speculation as to why Apple would roll this surveillance tool out, as a compromise to having a fully encrypted system.
> Do you have an actual suggestion that you think will cut down on CSAM beyond providing a redundant reporting system?
The reporting system isn't redundant, but beyond that, the impetus isn't on the person whose rights are being violated to offer solutions to replace that violation.
The law isn't going to rely on people that might or might not report CSAM if they see it. Average users have no obligation to report CSAM, unlike the law and the companies that store user data. If the law finds CSAM themselves, they will always report it, and in their eyes that only improves the chances of a successful conviction.
> My right to privacy is not an infringement of your rights so this argument has no bearing in reality.
The law would argue that a right to privacy that includes the ability to store CSAM privately is infringing on the rights of others, because allowing CSAM to be consumed and distributed creates a market that incentivizes the further abuse of children. CSAM cannot be produced without CSA taking place, which makes it a part of that cycle.
> There is no technical reason that this is required to enable E2EE for iCloud, it's purely speculation as to why Apple would roll this surveillance tool out, as a compromise to having a fully encrypted system.
That is Apple's own fault for failing to precisely explain their reasoning for designing the system the way they did, and the confusion was entirely preventable.
The reason this is murky territory is because it's on-device, and we have precedence of there being an expectation of privacy on our cell phones via Carpenter v. United States which found that in a very narrow circumstance third party doctrine did not apply. However, other cases have not been tested and there are many factors regarding Apple's implementation that may or may not protect it in an actual case. We don't really have a clear picture of how and when third party doctrine applies with the case of our cell phones, which have been determined to be nearly indispensable in modern society.
But there's a difference between the legality of something and morality to it and while the former is up in the air, the latter really isn't in this case, at least for a lot of people that are arguing for or against this system. In my opinion, it does not matter that Apple's implementation only ever activates if you there is intent to upload to their servers. The fact that it is performed on device is in itself the most problematic aspect. There are no acceptable justifications to be had here. If Apple wants to scan their own servers that is their prerogative. The same cannot be said about my phone.
You said you didn't want Apple to be the police but you know want them to be the judge of what is and isn't CSAM?
>It can also create tools to reduce the overhead of reporting. If you make it easier to report, more people will do it. You're assuming that anyone that ever encounters CSAM would go out of their way to report it, which simply isn't true.
I don't know why you are assuming that potential reporters knowing about CSAM but not reporting it is anywhere close to as common as CSAM being unknown and therefore obviously unreported.
>My right to privacy is not an infringement of your rights so this argument has no bearing in reality. Law enforcement requires probable cause to get permission to surveil the population and there's been countless cases thrown out because of this violation, which, as it turns out, was even tested in the context of technology interfacing with NCMEC.
As the other reply stated, certain content does infringe on other people's rights. Also Apple isn't law enforcement. They don't need probable cause. One of the primary and unmentioned motivators here is that they don't want CSAM ending up on their servers and opening themselves up to legal action.
>Those people are also pushing the same false dichotomy that you are. There is no technical reason that this is required to enable E2EE for iCloud, it's purely speculation as to why Apple would roll this surveillance tool out, as a compromise to having a fully encrypted system.
It is a technical requirement once you accept the moral requirement that Apple doesn't want to enable the sharing of CSAM. Once again, you are ignoring that some people think the morality of enabling (or at least not curtailing) the sharing of CSAM is a moral failing that can't be accepted.
Your argument of it being a moral requirement is dubious. If you really wanted to stop CSAM why not we all live in glass apartments like in Zamyatin's We? To what extent would invading your privacy be acceptable to you if it means we could stop some or even all CSAM distribution and production? Once you decide that it's a moral imperative then you accept that any erosion of privacy in the interest of saving the children isn't off the table.
There's no practical way to test this because there's no incentive for any of those companies to change how they scan for CSAM. None of those companies are going to take the risk of implementing something that could enable future sexual abuse crimes to take place just to improve privacy. The system in place apparently works, but there appears to be little statistical data validating those techniques.
What I would want to know is what alternative method for reporting CSAM could be developed that would meet the current expectations of the law in the number of cases they successfully uncover while also satisfying the desire for absolute privacy under all circumstances. A system that has users reporting content themselves relies on the users acting to incriminate other people, and the criminals can simply improve their privacy by hiding the content out of public view. It also doesn't address the fact that the government doesn't want people storing CSAM on their phones at all, and if they happen to notice you're storing CSAM for any reason, they're going to arrest you for it.
I'm honestly not sure that it's possible for any company to implement true E2EE given the current state of the law. It sounds like a tradeoff had to be made between privacy and the ability for the law to be upheld.
Isn't this in itself a win for reducing distribution of CSAM? The content is already out of public view. You can't just go to some clear web website and find CSAM unless you're incredibly unlucky and all legal entities have systems in place to report it if it does end up on there. Companies like pornhub have fairly recently created systems to reduce the chance of CSAM ending up on their platform by requiring verification. There's certainly nothing stopping most CSAM from being distributed encrypted through official or illicit channels and the sad reality is there probably is significantly more that is than isn't.
Law enforcement has a big task on their hands here and I can empathize with how difficult it can be but you cannot sacrifice an entire population's rights to catch the relatively few bad actors. This erosion of rights was justified after 9/11 to implement the PATRIOT Act and people are trying to use CSAM as a justification to do so now. It's not good.
> I'm honestly not sure that it's possible for any company to implement true E2EE given the current state of the law. It sounds like a tradeoff had to be made between privacy and the ability for the law to be upheld.
There's already companies that do offer E2EE encryption and there is no law requirements to scan for CSAM in the US. I'll concede the point that Google, Microsoft, and Apple are going to be under much more scrutiny of law enforcement and the government than a small privacy focused cloud storage provider, but the point stands that there is no legal reason they must implement this.
Making CSAM illegal isn't merely about preventing the material from being publicly visible. Having a ring of child abusers privately circulating CSAM outside of clearnet by using Tor or by exchanging thumb drives is just as damaging of an act to law enforcement agencies, because it means the market can still exist in a different environment. There will always be other ways to distribute CSAM, but what the law sees as within its reach in order to prevent the spread of CSAM to the greatest extent possible is to pressure companies that offer user data storage into increasing the amount of scanning for such material.
> you cannot sacrifice an entire population's rights to catch the relatively few bad actors.
Unfortunately this is a matter of public opinion, and there are many people out there that would argue instead that we need to give up some amount of privacy in order to better protect the human population.
> There's already companies that do offer E2EE encryption and there is no law requirements to scan for CSAM in the US.
What are the names of these companies? Are you sure there isn't a clause in their terms of service stating they will give up your data to law enforcement if requested, including the contents of encrypted files?
Even though there is no law compelling companies to scan for CSAM, the law is not going to let any of them store CSAM for an indefinite period of time. That would defeat the entire purpose of outlawing CSAM to begin with. Regardless of what's legally required of them, it's in the companies' best interests to ensure that the data stays off their servers. No matter what moral stance they take in deciding to implement CSAM scanning, they're not going to take the risk of being found criminally liable for the data they're storing.
How do you know what "the law" sees as being within its reach? Law enforcement and rights are always going to be at odds with each other.
> Unfortunately this is a matter of public opinion, and there are many people out there that would argue instead that we need to give up some amount of privacy in order to better protect the human population.
I don't really care so much about this hypothetical public opinion. Refrain from using this unknowable nebula of consensus to argue your points for you. Even if there is massive consensus that would not change my argument that it is wrong.
If a company has no means of which to know what is stored on their servers, they have no reason to presume that it is illicit material, even if it in fact is. Removing those liabilities by restricting what Apple can know has been how Apple's been operating for quite some time. Even this implementation could be argued that this is what Apple is attempting to do. The issue is that ultimately, Apple is installing software on your device to work only as adversary to its user by violating their privacy.
> What are the names of these companies? Are you sure there isn't a clause in their terms of service stating they will give up your data to law enforcement if requested, including the contents of encrypted files?
By definition if the third party can decrypt your files it is not end to end encrypted, the third party in this case being Apple. Many files upload to iCloud already are encrypted but since Apple also has access to the key it can decrypt and examine the contents of the files on its servers. With E2EE the only person with access to the private key would be yourself. They could certainly hand over the encrypted files and if law enforcement gained access to your private key they could then decrypt those files, but (hypothetically) not until after they've obtained the private key.
In my mind, what might have actually transpired for Apple to announce this change was a less severe instance of such governmental or legal pressure on Apple to scan as much data as the other major tech companies. Last year Apple only submitted a couple hundred reports to the NCMEC, while Facebook submitted 20 million. Looking at those numbers makes it sound like Apple was a nail sticking out that needed to be hammered down.
How does that change the warrant's effectiveness? Should you be obligated to unlock your phone for authorities? Otherwise the warrant still does nothing because the device's encryption can't be broken without forcing a person to potentially self-incriminate themselves. This is fundamentally the same problem that encryption breaks how warrants used to behave.
So now they have a wedge that they can gradually use to expand what is going to be scanned and when.
This is a classic power move straight from 48 laws of power.
Speaking out enough on this gets you called a pedo.
I will not go as far to claim that the people for this are intentionally using children to shield themselves from criticism push out a universal surveillance mechanism. ... but if you wanted to do exactly that, hiding behind the spectre of child abuse is exactly that.
There are some hard questions we should be asking. Facebook claims to have reported some 20 million insances of child porn last year, but there were only a couple prosecutions. Are the matches fake? Is the goal really to just build a huge database of kompromat? If these images aren't a concern why are only an infinitesimal proportion of cases prosecuted?
I think this is actually a pretty weak argument. Children may be vulnerable, but so are many other segments of the population, including those with disabilities, the elderly, and so on. I guess my question is if we're trying to protect children, then why not go all the way? The answer is because going all the way is dangerous: where does it stop? Why here but not there?
I think a better argument would be: we should scan phones for X because X is illegal, and we've deemed, as a society, that X is wrong/bad/punishable/etc. But the problem with that argument is that we already have ways of checking phones/user data for X, and, thankfully, most have to go through a legal process of probable cause, obtaining warrants, and so on. Apple's proposal completely circumvents the legal system.
This is a complicated question that I'm sure will be litigated in court in the next few years. SCOTUS has ruled that drug sniffing dogs (e.g. in airports) are constitutional, whereas using infrared imaging (e.g. for finding drug grow houses) is not. Are hashing algorithms more like drug sniffing dogs or like infrared imaging?
But if the lack of a warrant is the problem then the main issue would be the third-party doctrine and having no reasonable expectation of privacy when entrusting your data with any company, not just Apple.
Because, if I were to compare to what OP claims here, constant device scanning and then contacting law enforcement is like having a company constantly infrared scan your house for drug growing, and then contact the authorities about it. This is circumventing the authorities' legal limitations by moving the burden of responsibility to a company.
Google, Facebook, and others are already scanning the images when uploaded to them. We should not be surprised that the images we upload are being scanned. Sometimes for CSAM, sometimes to train ML, and sometimes for other things.
Apple doing scanning images that were uploaded to their servers would just be another company doing the same thing.
If I understand things correctly, these companies even have some legal obligations to do some of this.
> However, I believe that they should not run the scan on the device, because that carries significant drawbacks for personal privacy.
This is about where Apple is doing the scanning. Doing it in icloud is on them. Doing it on our phones is an invasion of privacy. And, it builds a system that can be used for other things. It opens a door, as many others have pointed out.
I wonder if Apple did all of this to scale the implementation. So it wouldn't have to run on their servers. They wouldn't have to maintain and power them. Poor implementation if that's the case.
Should GM use onstar to record every conversation you have in your car and use AI to flag conversations that relate to child abuse? After all it involves children.
And if down the road law enforcement decides to expand what gets flagged, you shouldn’t care if you aren’t breaking any laws, right?
This is a horrible idea and it’s a matter of if, not when it is abused.
Maybe if masses of people go into apple stores and save photos on demo phones that collide with the files they're looking for. Do those phones get scanned? Will apple ban their own app store accounts?
I guess it depends on whether the system is 100% safe and cannot be exploited;
Since only iCloud photos are scanned, and other cloud storage providers already do this and have been scanning their users photos for a long time. I haven’t heard anyone complain when Google started scanning their cloud photos, which is kind of… interesting.
The data is not what causes the harm. It is the method used to produce (some of) the data that causes the harm.
And the problem is that looking for the data in order to find the people causing the harm runs afoul of the converse fallacy. It's the same fallacy that causes people to think that because most terrorists (that they know of) are Muslims, that most Muslims must be terrorists. Most child abusers have photos of naked children on their phones, therefore most people who have photos of naked children on their phones must be child abusers. It's false in both cases. A friend of mine, for example, has photos of her naked child on her phone that she sent to the doctor to help diagnose a rash.
The legal system already provides a means for invading people's privacy given certain criteria, whether by scanning their phones, physically searching their residences and offices, or whatever. The criterion is "probable cause" and the search is launched by a judge signing a warrant that the probable cause exists. That is how the Framers wrote it into the US Constitution (see Amendment 4), because they had dealt with too many bogus searches done by the Colonial governments and wanted to put a stop to that. Things have not really changed since then.
"Think of the children!!!" is a traditional thought-stopper, but really, if there is substantive reason to search someone's phone, just convince a judge to sign a warrant and everything will be fine. Searching people's stuff when they are not under any concrete suspicion is part of why we had a revolution in this country in the first place. I.e., why does Apple hate America, as the memes like to put it?
If your problem with Apple's proposal is the fact they do hash matching (rather than the system is run on your device), why is the criticism reserved for Apple instead of being directed at everyone who does hash matching to find CSAM? It seems like a lot of the backlash is because Apple is being open and honest about this process. I worry that this will teach companies that they need to hide this type of functionality in the future.
If you're concerned about ways Apple might access on-device contents in the future ... there's myriad ways for your OS vendor to circumvent the privacy protections offered by your OS ...
[0] or is at least as comprehensive a mitigation as possible on systems with access to cleartext
I don't think the hashing exacerbates the problem because the alternative to Apple's proposal seems like it would be to run the same hash matching on the server like those other companies. That doesn't fix the hash collision problem.
I completely agree that the magnitude of the backlash against Apple is not fair in comparison to the backlash against every other tech company using PhotoDNA, and it feels hypocritical to for everyone to have given them a pass for over a decade despite the principle and intent remaining the same.
It would be like if the bank searched your safe deposit box stored in their vault in the course of their business, but then later announced that to reduce the need to search your vault they'd be stationing a spy in your home to inspect the box before you leave for the bank. The spy will secretly report back to the bank if he sees something he suspects, and then the bank will search.
With these collision examples, you now learn that the spy will also be tripping balls.