ImageNet contains naturally occurring Apple NeuralHash collisions
blog.roboflow.com
blog.roboflow.com
Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar).
In an ordinary, non-adversarial context, the probability that you win the game is much less than one in one trillion, so this game is very safe for the casino. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play, and the relevant probability is no longer one in one trillion. The relevant number is 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino.
Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.
From what I see, the end result of false flagging is either someone has CSAM in iCloud and you push them over the threshold that results in reporting and prosecution, or there is no CASM, so the reviewer sees all of the hash collision images, including those that are natural.
Is the problem that an attacker can force natural hash collision images to be viewed by a reviewer, violating that persons privacy? Do we know if this process is different than how Google, Facebook, Snapchat, Dropbox, Microsoft, and others have implemented these necessarily fuzzy matches for their CSAM scans of cloud hosted?
Or am I missing something that the downvoters saw?
If your account has a large number of safety vouchers that trigger a CSAM match, then Apple will gather enough fragments to reassemble a secret key X (unique to your device) which they can use to decrypt the "visual derivatives" (very low resolution thumbnails) stored in all your matched safety vouchers.
An Apple employee looks at the thumbnails derived from your photos. The only judgment call this employee gets to make is whether it can be ruled out (based on the way the thumbnail looks) that your uploaded photo is CSAM-related. As long as the thumbnail contains a person, or something that looks like the depiction of a person (especially in a vaguely violent or vaguely sexual context, e.g. with nude skin or skin with injuries) they will not be able to rule out this possibility based on the thumbnail alone. And they will not have access to anything else.
Given the ability to produce hash collisions, an adversary can easily generate photos that fail this visual inspection as well. This can be accomplished straightforwardly by using perfectly legal violent or sexual material to produce the collision (e.g. most people would not suspect foul play if they got a photo of genitals from their Tinder date). But much more sophisticated attacks [2] are also possible: since the computation of the visual derivative happens on the client, an adversary will be able to reverse engineer the precise algorithm.
While 30 matching hashes are probably not sufficient to convict somebody, they're more than sufficient to make somebody a suspect. Reasonable suspicion is enough to get a warrant, which means search and seizure, computer equipment hauled away and subjected to forensic analysis, etc. If a victim works with children, they'll be fired for sure. And if they do charge somebody, it will be in Apple's very best interest not to assist the victim in any way: that would require admitting to faults in a high profile algorithm whose mere existence was responsible for significant negative publicity. In an absurdly unlucky case, the jury may even interpret "1 in 1 trillion chance of false positive" as "way beyond reasonable doubt".
Chances are the FBI won't have the time to go after every report. But an attack may have consequences even if it never gets to the "warrant/charge/conviction" stage. E.g. if a victim ever gets a job where they need to obtain a security clearance, the Background Investigation Process will reveal their "digital footprint", almost certainly including the fact that the FBI got a CyberTipline Report about them. That will prevent them from being granted interim determination, and will probably lead to them being denied a security clearance.
(See also my FAQ from the last thread [1], and an explanation of the algorithm [3])
[1] https://news.ycombinator.com/item?id=28232625
[2] https://graphicdesign.stackexchange.com/questions/106260/ima...
> Given the ability to produce hash collisions, an adversary can easily generate photos that fail this visual inspection as well.
Apple could easily fix this by also showing a low-res version of the CSAM image that was collided with, but I'll grant that they may not be able to do that legally (and reviewers probably don't want to look at actual CSAM).
The problem isn't CSAM scanning in principle. The problem is that the shift to the client & the various privacy-preserving steps Apple is attempting to make is actually making the actions taken in response to a match different in a concerning way. One big problem isn't the cases where the authorities should investigate*, but that a malicious actor can act surreptitiously and leave behind almost no footprint of the attack. Given SWATting is a real thing, imagine how it plays out if child pornography is a thing. From the authorities perspective SWATting is low incidence & not that big a deal. Very different perspective on the victim side though.
[1] https://embracethered.com/blog/posts/2020/husky-ai-image-res...
* One could argue about the civil liberties aspect & the fact that having CSAM images is not the same as actually abusing children. However, among the general population that line of reasoning just gets you dismissed as supporting child abuse & is only starting to become acknowledged in the psychiatry community.
Guess what, they can already do this but worse by just sending you actual illegal images of 17.9 year olds.
While it would be bad to be subjected to such an attack, and there is a small chance it would lead to some kind of interaction with law enforcement, the outcomes you present are just scaremongering and not reasonable.
The point is precisely that people can become victims of various new attacks, without ever touching photos that are actual "known CSAM". For Christ's sake, half the comments here are about how adversaries can create and spread political memes that trigger automated CSAM filters on people's phones just to "pwn the libz".
> Guess what, they can already do this but worse by just sending you actual illegal images of 17.9 year olds.
No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people. E.g. your own photos are not in the NCMEC databases, and you'd have to reveal your own illegal activities to get them in there. You (or malicious political organizations) especially cannot attack and expose "wrongthinking" groups of people by sending them photos of 17.9 year olds.
> "No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people."
Did I say "taking"? I am talking about sending (theoretical) actual images from the NCMEC database. This is functionally identical to the "attack" you describe.
You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent image. Note that knowing such a hash and having such an image are both perfectly legal. Moreover, since the resizing (the creation of the visual derivative) is done on the client, you can tailor your scaling attack to the specific resampling algorithm.
Eventually, someone will make a CyberTipline report about the actual CSAM image whose hash you used, and the image (being a genuine CSAM image) will make its way into the NCMEC hash database. You will even be able to tell precisely when this happens, since you have the client-side half of the PST database, and you can execute the NeuralHash algorithm.
You can start circulating the meme before or after this step. Repeat until you have circulated enough photos to make sure that many people in the targeted group have exceeded the threshold.
Note that the memes will trigger automated CSAM matches, and pass the Apple employee's visual inspection: due to the safety voucher system, Apple will not inspect the full-size images at all, and they will have no way of telling that the NeuralHash is a false positive.
[1] https://github.com/anishathalye/neural-hash-collider
[2] https://embracethered.com/blog/posts/2020/husky-ai-image-res...
[3] https://bdtechtalks.com/2020/08/03/machine-learning-adversar...
[4] https://graphicdesign.stackexchange.com/questions/106260/ima...
(Perhaps I misunderstood your original post, but this seems to be a completely different scenario to the one you originally described with reference to the three thumbnails)
1. Get a photo that the target already has.
2. Generate an objectionable image with the same hash as the target's photo. (This is obviously illegal.)
3. Submit the objectionable image to the government database.
Now the target's photo will be flagged until manually reviewed.
This doesn't sound impossible as a targeted attack, and if done on a handful of images that millions of people might have saved (popular memes?) it might even grind the manual reviews to a halt. But maybe I'm not understanding something in this (very bad idea) system.
The point though is that this is something someone's Apple phone is doing, that their device is not. So the goal is to send a hash collided images by non-Apple channels (email) where there is a reasonably good chance that image would make it's way into someone's global device photo store and into automatic iCloud uploads.
Sending an MMS would work, for example, or a picture to Signal which then someone saves to outside of Signal (a meme).
In all these cases, the original sender doesn't have an Apple device: so they're not getting scanned by the same algorithm, but more importantly their device is not spying on them. Importantly too: they've done nothing illegal.
But: the victim is getting flagged by their own device. And the victim has to have their device seized and analysed to determine (1) that it's not CSAM, (2) that they were sent those images that flagged and aren't trying to divert attention by getting themselves false pinged upfront, but then (3) the sender has committed no crime. There's no reason or even risk to investigate them, because by the time the victim has dealt with law enforcement, it's been established that no one had anything illegal.
It's the digital equivalent of a sock of cat litter testing positive as being methamphetamine, except if it was your drive through McDonald's order.
The goal is not to get convictions, the goal is harrassment.
> they've done nothing illegal
Perhaps that's true in the narrowest sense, but aren't the odds of generating a colliding file so low as to all but rule out coincidence and therefore strongly indicate premeditated cyber-attack (which is illegal)?
If I were law enforcement, at the very least I'd want to keep tabs on these sources of false positives. Probably easy enough to convince a judge that someone capable of the "tech wizardry" to collide a hash can un-collide one too, and therefore more thorough/invasive search warrants of the source are justified.
Like security researchers. Or the people currently inspecting the algorithm. And also frankly what are you going to do about overseas adversaries? The most likely people looking at how to exploit this would explicitly be state-sponsored Russian hackers - this is right up the alley of their desire to be able to cause low level chaos without committing to a serious attack.
And at the end of the day you've still succeeded: the point is that by the time you've established it was spurious, the target has already been through the legal wringer. The legal wringer is the point.
(Sorry, this is going to be long. I know understand most/all of this stuff, it's mostly there to provide a bit of context for the users reading our exchange)
The term "hash function" is a bit of a misnomer here. When people hear "hash", they tend to think about cryptographic hash functions, such as SHA256 or BLAKE3. When two messages have the same hash value, we say that they collide. Fortunately, cryptographic hash functions have several good properties associated with them: for example, there is no known way to generate a message that yields a given predetermined hash value, no known way to find two different messages with the same hash value, and no known way to make a small change to a message without changing the corresponding hash value. These properties make cryptographic hash functions secure, trustworthy and collision-resistant even in the face of powerful adversaries. Generally, when you decide to use two unrelated cryptographic hash algorithms instead of one, executing a preimage attacks against both hashes becomes much more difficult for the adversary.
However, as you know, the hash functions that Apple uses for identifying CSAM images are not "cryptographic hash functions" at all. They are "perceptual hash functions". The purpose of a perceptual hash is the exact opposite of a cryptographic hash: two images that humans see/hear/perceive (hence the term perceptual) to be the same or similar should have the same perceptual hash. There is no known perceptual hash function that remains secure and trustworthy in any sense in the face of (even unsophisticated) adversaries. In particular, preimage attacks against perceptual hashes are very easy, compared to the same attacks against cryptographic hashes.
Using two unrelated cryptographic hashes meaningfully increases resistance to collision and preimage attacks. Using ROT13 twice does not increase security in any meaningful sense. Using two perceptual hashes, while not as bad, is still much closer to the "using ROT13 twice for added security" than to the "using multiple cryptographic hashes" end.
Finding a SHA1 collision took 22 years, and there are still no effective preimage attacks against it. Creating the NeuralHash collider took a single week. More importantly, even if you were to use two unrelated perceptual hash functions, executing a preimage attacks against both hashes need not become much more difficult for the adversary: easy * easy is still easy. Layering cryptography upon cryptography is meaningful, but only as long as one of the layers is actually difficult to attack. This is not the case for perceptual hashes. In fact, in many similar contexts, these adversarial attacks tend to transfer: if they work against one technique or model, they often work against other models as well [3]. In the attack discussed above, the adversary has nearly full control over the "visual derivative", so even a very unsophisticated adversary can subject the target thumbnail itself to the collider before performing the resizing attack, and hope that it transfers against the second hash. If the second hash is a variant of NeuralHash (somewhat likely, it could even be NeuralHash performed on the thumbnail itself; we don't know anything about it!), or if it's a ML model trained on the same or similar datasets (quite likely), or if it's one of the known algorithms (say PhotoDNA) then some amount of transfer is likely to happen. And given an adversary that is going to distribute a large number of photos anyway, a 10% success rate is more than enough. Given the diminished state space (fixed size thumbnails, almost certainly smaller than 64x64 for legal reasons), a 10% success rate is completely plausible even with these naive approaches. An adversary that has some (even very little information) about the second hash algorithm can do much more sophisticated stuff, and perform much better.
But what if we boldly rule out all transfer results? Doesn't Apple keep their algorithm secret?! Can we think of the weights (coefficients) of the second perceptual hash as some kind of secret key in the cryptographical sense? Alas, no. Apple would have to make sure that all the outputs of the secret perceptual hash are kept secret as well. Due to the way perceptual hashing algorithms work, they provide a natural training gradient having access to sufficiently many input-outputs examples is probably enough to train a high-fidelity "clone" that allows one to generate adversarial examples and perform successful preimage attacks even if the weights of the clone are completely different from the secret weights of the original network. This can be done with standard black box techniques [4]. It's much harder (but nowhere near crypto hard, still perfectly plausible) to pull this off when they have access to one bit of output (match or no match). A single compromised Apple employee can gather enough data to do this given the ability to observe some inputs and outputs, even if said employee has no access to the innards or the magic numbers. The hash algorithm is kept secret because if it wasn't, an attack would be completely trivial: but an adversary does not need to learn this secret to mount an effective attack.
These are just two scenarios. There are many others. "Nobody has ever demonstrated such an attack working end-to-end" is not a good defense: it's been two weeks since the system was rolled out, and once an attack is executed, we probably won't learn about it for years to come. But the attacker can be rewarded way before "due process" kicks in: e.g. if a victim ever gets a job where they need to obtain a security clearance, the Background Investigation Process will reveal their "digital footprint", almost certainly including the fact that the NCMEC got a report about them, even if the FBI never followed up on it. That will prevent them from being granted interim determination, and will probably lead to them being denied a security clearance. If you pull off this attack on your political opponents, you can prevent them from getting government jobs, possibly without them ever learning why. And again, this is one single proposed attack. There were at least 6 different attacks proposed by regular HN users in the recent threads!
As a more general observation, cryptography tends to be resistant to attacks only if one can say things such as "the adversary cannot be successful unless they know some piece of information k, and we have very good mathematical reasons (e.g. computational hardness) to believe that they can't learn k". The technology is flawed: even the state-of-the-art in perceptual hashes does not satisfy this criterion. Currently, they are at best technicool gadgets, but layering technicool upon technicool cannot make their system more secure.And Apple's system is a high-profile target if there ever was one.
Barring a major breakthrough in perceptual hashing (one that Apple decided to keep secret and leave out of both whitepapers), the claim that the secret second hash will prevent collision attacks is not justified. The chances of such a secret breakthrough are very slim: it'd be like learning that SpaceX has already built a base on the Moon and has been doing regular supply runs with secret spaceships. Vaguely plausible in theory (SpaceX has people who do rocketry, Apple has people who do cybersecurity), but vanishingly unlikely in practice.
And that's before we mention that the mere existence of the collider made the entire exercise completely pointless: the real pedos can now use the collider to effectively anonymize their CSAM drops, making sure that all of their content collides with innocnent photos, and ensuring that none of the images will be picked up by NeuralHash anyway. For all practical purposes, Apple's CSAM detection is now _only_ an attack vector, and nothing else.
[3] https://arxiv.org/abs/1809.02861 [4] https://towardsdatascience.com/adversarial-attacks-in-machin...
For the second part, yes if an Apple engineer (that had access to this code) leaked the internal hash function they used or a bunch of example image’s to hash values, that would allow these adversarial attacks.
Until you can show an example or paper where the same adversarial image generates a specific hash value for two unrelated perceptual hash functions, with one being hidden, it is not right to predict a high likelihood of that first scenario being possible.
Here’s a thought exercise, how long would it have taken researches to generate a hash collision with that dog image if the NeuralHash wasn’t public and you received no immediate feedback that you were “right” or getting closer along the way?
[1] https://www.apple.com/child-safety/pdf/Security_Threat_Model...
"There is no paper attacking ROT13 done twice, therefore it must be secure". Usually, it's on the one proposing the protocol to make a case for its security. Doubly so when it's supposed to last a long time, a lot of people are interested in attacking it, and successful attacks can put people in harm's way.
You know what, if you think that this is difficult, feel free to pick an existing perceptual hash function H, cough up some money, and we'll announce a modest prize (say $4000) on HN for the first person to have a working collision attack for NeuralHash+H. H will run on a scaled-down thumbnail, and we'll keep the precise identity of the algorithm secret. If the challenge gets any traction, but nobody succeeds within 40 days, I'll pay you $4000 for your effort. If you're right, this should be easy money. (cf SHA1, which lasted 22 years)
Heck If Apple claims that this is difficult (afaict they don't, it would be unwise), they might even join in with their own preimage challenge for $$$. It'd be a no-brainer, a simple and cheap way of generating good publicity.
If I took an exact public perceptual hash function implementation and used that as H in your contest, it might be possible for a researcher attacking all public perceptual hash functions to stumble on the right one within 40 days.
I agree with you that we are trusting Apple to implement this competently. This isn’t something that can be proved to work mathematically where nothing about the implementation has to be kept secret.
So worse case everything you say could come true but to imply that is likely is wrong.
An attacker can embed a matching image inside of a PowerPoint zip file, and email it to any corporate employee using O365.
Or, an angry parent can call the police and let them know that a 16 year old possesses nose pictures of their 15 year old girlfriend.
The over top response to this controversy is really disappointing.
For one last time, the NeuralHash collisions make this tool perfectly unusable for catching pedos: all of the next generation of CSAM content will collide with hashes of popular, innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. It's completely useless for its nominal function. This would be a massive, hilarious own goal from Apple even if the public reaction was over the top (although it isn't). They just reduced the privacy and security of nearly all their customers, further exposed themselves to the whims of governments, and for no gain whatsoever.
This can be abused to spam Apple's manual review process, grinding it down to a halt. You've cost Apple time and money by making them review each such fake report.
Ok, but… how do I profit? If I wanted to waste Apple employee time, I could surely find a way to do it, but why would I? The functioning of society relies on the fact that people generally have better things to do than waste each others time.
To some people the ethical problems with this system are worth the effort to make their implementation impossible.
if you are asking for monetary profit, that does not have to be a goal for someone to want to engage in disruption of this service.
For the lulz.
You can't rely that it won't ever leak, and when it does, it will be almost undetectable and have huge consequences.
As soon as the first on-device CSAM flag has been raised, it becomes a legal and political problem. Even without a second matching hash, it already put Apple in an untenable position. They already are in a mud fight with the pigs.
They can't say : we got 100M hits this month on our first CSAM filter but we only reported 10 cases, because to avoid false positives our second filter throw everything to dev/null, and we didn't even manually reviewed them because your privacy matter to us. It has become a political problem where for good measure they will have to report cases to make the numbers look "good".
Attackers of the system can also plant false negatives aka real CSAM that has been modified enough to pass the first hash but fail this second hash. So that, in the audit, independent security researchers who review Apple system, will be able to say that Apple automated system, sided with the bad guys, by rejecting true CSAM and not reporting it.
Also remember, that Apple can also do something else than what they say they do for PR reasons : maybe some secret law will force them to reveal to the authorities as soon as the first flag has been raised, and force them not tell about it. And because it's in the name of fighting the "bad guys", that's something most people expect them to do.
From the user perspective, there is nothing we can audit, it's all security by obscurity disguised with pseudo-crypto-PR, it's just a big "Trust us" blanked signed paper that will soon be used to dragnet surveil anyone for any content.
Because the perceptual hash algorithms are presented as black boxes the image they perceive isn't audited or reviewed. There's zero recognition of this weakness by Apple or NCMEC (and their equivalents). For the system to even begin to be trustworthy all content would need to be reviewed raw and scaled-as-fed-into-the-algorithm.
[0] https://bdtechtalks.com/2020/08/03/machine-learning-adversar...
Granted, Apple makes no mention of any safeguard, but it would be trivial in principle to protect against, and is not an unavoidable failing.
You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database.
Note its NeuralHash.
Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination.
Feed that imagine into an adversarial network until it spits out the desired NeuralHash.
Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.
Yeah, mentioned something like that here [0]:
>> And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this.
With that out of the way, that, obviously, just forecloses this one particular attack, specifically, where you want the thumbnail to appear dramatically different than the full image in order to fool the user that it's an innocent image and the reviewer that it's an illegal image. It's still, never-the-less, possible to have a confusing thumbnail -- perhaps an adult porn image engineered to have a CSAM hash collision will be enough to convince a beleaguered or overeager reviewer to pull the trigger. The "Image Scaling Attack" is neither sufficient or necessary.
(However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review, so would never be shown to a human reviewer: "as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash" [1])
[0] Understanding and Preventing Image-Scaling Attacks in Machine Learning https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf
[1] https://www.apple.com/child-safety/pdf/Security_Threat_Model...
Uh, on what timescale? If you mean "tomorrow" then sure, if you mean "for years" - then no. They're relying on the second perceptual hashing algorithm to remain a secret, which is insanely foolish. Just based on what I know about these CP hashlists and the laziness of programmers, I feel pretty confident that it is either an algorithm trained on the thumbnails themselves (which would be laughably bad) or it was a prior attempt that got replaced by what is now deployed on the users' hardware. Why would I think that? Because it would have been the only other thing on hand for the necessary step of generating the hash black list. So they're stuck with at least one of those forever - and will have a very limited range of potential responses to the massive infosec spotlight picking them apart... unless they want to recatalog every bit of CP all over again.
While there may not be an immovable obstacle standing between them and a complete recataloging, there are a lot of factors that would strongly disincentivise it. Chief among them being the fact that the project is a already a radioactive cost center - and unless they plan on switching industries and giving Blue Coat a run for its money, it always will be.
> ...chaining together multiple hashes.
That would be the lazy programmer way to do it that would very likely result in a situation where correlation starts popping up - that is why DBAs weren't advised to do some whacky md5/sha1 mashup that avoids requiring every user rekey in the wake of a digest bump up.
> ...I would guess that they would just use Microsoft’s PhotoDNA hash unchanged...
That is a reasonable guess, because that is what all the NGOs have been using - IWF being one of the more notorious. That would be bad news though, for anyone expected the thumbnail perceptual hashing step to provide meaningful protection.
> I would think using a completely different technology would make the pair of algorithms extremely robust...
Nope - which is why you don't see hybrid cryptographic algorithms. Also, if they are using PhotoDNA on their verification step then they actually implemented the thing totally backwards... because the high-pass filter approach makes it resistant to the hash length extension attacks that are imperceivable to humans. That counts for nothing by the time the first algorithm has been fooled by an extension attack (and this neural thing is definitely vulnerable to it), because the attacker would already be selecting for a thumbnail image that would fool a human in the second step - and PhotoDNA would be looking for the exact same thing that a human would: points of contrast.
BTW, PhotoDNA is a black box with no outside scrutiny to speak of - you can count on one hand the number of papers where it is even mentioned (and only ever in passing).
And if you can do that, why wait for apple to implement this? Why not just put child porn on their phone and then call the police?
That does not scale.
> … why wait for apple to implement this?
Enables scaling of what the above commentator discusses.
I have a hard time seeing this happen with a "perfect" scaling algoritm in the spatial domain.
Wouldn't that mean that the system can be poisoned? How would they deal with that?
Other than that, yes it's possible to add noise to an image so a perceptual algorithm misidentifies it. I described the false positive case but it can also be used for false negatives. Someone can apply noise to a legit CSAM image (in the NCMEC database) so Apple's system fails to identify it.
The false positive case is scary because if it happens to you your life is ruined. The false negative case just means people have CSAM and don't get found by the system. I'm much more concerned about the false positive case.
Eventually, these photos will make their way into the NCMEC database, and produce a large number of false positives. This will also make the other attacks discussed here easier to execute (e.g. by lowering the human review threshold, since everybody will start with a few strikes).
As others said, if the non-csam looks sexual at all, they'll probably get flagged for post-apple review.
Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives.
As many mentioned, even an investigation can have horrible affects on some (innocent) person's life. I would not be shocked to learn that some crafty individual working at "meme factories" creating intentional collisions with distributed images just for "fun" - and politically motivated attacks seem plausible (eg. make liberal political memes flag CSAM).
Then there are targeted motives for an attack. Have a journalist you want to attack or find a reason to warrant? Find them on dating app and send them nudes with CSAM collisions. Or any number of other targetted attacks against them.
I would have thought the opposite. If there is a false positive that leads to an arrest and ruins someone's life, but the public sees that it is a false positive, then Apple will take an enormous hit in the marketplace. Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM.
If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple.
> Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM.
I don't think this is much of a risk. Defamation is difficult to prove, never applies to the law enforcement agencies who are going to make these arrests and "ruin people's lives", is never going to be down to Apple anyway (Apple is only referring things for investigation). I don't think you'll see much public indignance about this--even arguing against it in the real public eye (outside of wonky venues like HN) sounds tantamount to "supporting kiddie porn" in the naïve view of most members of the public.
> If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple.
This could potentially arise in any case with an abuser or producer or enabler or pimp or Maxwell/Epstein customer who has an iPhone; which is a lot of cases. As soon as Apple devices are supposed to "detect" kiddie porn, people will ask why people like this weren't caught earlier; and since Apple has money, they won't just ask this in the court of public opinion, they will sue for damages for abuses that "should have" been prevented by Apple's inspection of their pictures. Even if that's unlikely, it's much easier for a peon whose job it is to look at kiddie porn to just forward it on; and such a case really could damage Apple's optics.
The CASM thing, I guess, give at most a dozen a week...
2) modify close up/ambiguous adult porn to be flagged as CP with free GitHub tool
3) batch a few thousand porn photos like this to poison them
4) upload them everywhere, 4chan/Reddit/tumblr/discord/imagefap
5) some poor sap manages to save 20+ of your bait images
6) apple reviewer sees 100x100px blurry gray image of definitely porn that was flagged as CP. hits report.
7) a SWAT team bust down your door and takes all your devices while you go to jail and your mugshot is everywhere
Someone will do this, and the poisoned images will spread organically until they find a victim
At least I'm given to understand that they already scanned all these photos uploaded to iCloud anyway (in the same way many other similar providers do). Whether it happens on the device or the server doesn't seem to make any difference to this attack.
(That's not to say that (a) the scanning of stuff on a server was a good idea in the first place or (b) encouraging politicians to use your own device to spy on you is a good idea or (c) this isn't the thin end of a very painful wedge, just that we've not opened a new vulnerability)
Does that sound consistent with how Apple has positioned itself in the market?
* a call center agent at a haulage firm, instead of entering the delivery date we talked about on the phone, clicked for the delivery to be returned to the factory.
* Google automatically blocked an ad account from delivering ads because we allegedly profiteered from Covid (untrue of course, but we surly talked about the challenges caused by the pandemic somewhere on the site, so the "AI" apparently got triggered by some keywords), and humans repeatedly confirmed the AI decision.
* Facebook blocked an ad account that was unused in 2020, wanted ID, got the correct ID (identical name etc.), and the human denied confirmation.
Google and Facebook are of course known to be beyond kafkaesque, so this is no surprise. But imagine the costs the innocents pay once they accidently get entered into the FBI CP suspect database.
I don’t want there to be a chance some person is going to look at my pics!
To be clear - Apple claims that there is a one in a trillion chance of there being ~30 false matches in a single account.
> This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark.
It's great to see the ingenuity and attention this whole debacle is receiving from the community. Maybe it will lead to advances in perceptual hashing (and also advances in consumer awareness of tech related privacy issues).
Apple reported a pretty similar collision rate so maybe they did.
> In Apple’s tests against 100 million non-CSAM images, it encountered 3 false positives when compared against NCMEC’s database. In a separate test of 500,000 adult pornography images matched against NCMEC’s database, it found no false positives.
[1] https://tidbits.com/2021/08/13/new-csam-detection-details-em...
https://www.apple.com/child-safety/pdf/Expanded_Protections_...
Reporting the collision rate per image pair feels misleading. What you really want to know is the number of false positives per image in the relevant set, not image pair, as that's the figure that indicates how frequently you'll hit a false positive.
Why not?
Heck, how is it relevant to anything given that hash functions avoid collisions in practice, even given the pigeonhole principle?
> Perhaps the most concerning part of the whole scheme is the database itself. Since the original images are (understandably) not available for inspection, it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse.
Apple will also refuse all
requests to instruct human reviewers to file reports for
anything other than CSAM materials for accounts that exceed
the match threshold.
The only thing they will have gained access to are the “derivatives” (presumably lower res versions) of the matched photos, which if this is done to frame you is strictly the fake CSAM.
What do you mean
I live in one of those countries and I'd want to be aware
but actually this is a good starting point: https://en.wikipedia.org/wiki/LGBT_rights_in_Poland
It's not like China, or India, who not only have huge markets, but could easily hold a chunk of Apple's supply chain hostage.
It's very easy to uphold human rights if it doesn't actually cost you anything.
As someone not in the tech field, it is incredibly concerning that half the people here on Hacker News, people who help build this kind of technology, do not seem to be concerned with what Apple is doing.
That isn't a bug, it is a feature and will be the main use of this functionality.
The "preventing child pornography" reasoning was specifically chosen so that Apple could openly coordinate with governments to violate your privacy while avoiding criticism.
If you trusted Apple not to stealthily run such technology before, the question is how much less (if any) you trust them now.
If you didn’t, I don’t think anything changed.
They can't say if it is or not a match so they have to go after the individuals. Is that enough evidence for a warrant?
Someone in the court thinks it's true and can't prosecute?, oh, it got leaked.
--
Not every country has the same protections about innocent until proven guilty. And even then, we've seen cases in the US where someone has been held in jail indefinitely until they provide a password,
- https://arstechnica.com/tech-policy/2016/04/child-porn-suspe...
- https://nakedsecurity.sophos.com/2016/04/28/suspect-who-wont...
And yes, it's true that the governments could always mandate such scanning before. The difference is that it'll be much harder politically for Apple to push back against tweaks to the scheme (such as lowering the bar for manual review / notification of authorities) if they already have it rolled out successfully and publicly argued that it's acceptable in principle, as opposed to pushing back against any kind of scanning at all.
Once you establish that something is okay in principle, the specifics can be haggled over. I mean, just imagine this conversation in a Congressional hearing:
"So, you only report if there are 30+ CSAM images found by the scan. Does this mean that pedophiles with 20 CSAM images on their phones are not reported?"
"Well... yes."
"And how did you decide that 30 is the appropriate number? Why not 20, or 10? Do you maybe think that going after CSAM is not that important, after all?"
There's a very old joke along these lines that seems particularly appropriate here:
"Churchill: Madam, would you sleep with me for five million pounds?
Socialite: My goodness, Mr. Churchill… Well, I suppose… we would have to discuss terms, of course…
Churchill: Would you sleep with me for five pounds?
Socialite: Mr. Churchill, what kind of woman do you think I am?!
Churchill: Madam, we’ve already established that. Now we are haggling about the price."
Apple has put itself in the position where, from now on, they'll be haggling about the price - and they don't really have much leverage there.
Haven’t they always been able to do that?
>And yes, it's true that the governments could always mandate such scanning before. The difference is that it'll be much harder politically for Apple to push back against tweaks to the scheme (such as lowering the bar for manual review / notification of authorities) if they already have it rolled out successfully and publicly argued that it's acceptable in principle, as opposed to pushing back against any kind of scanning at all.
>Once you establish that something is okay in principle, the specifics can be haggled over. I mean, just imagine this conversation in a Congressional hearing:
Assuming said socialite was not in a committed relationship, why would they not take that money for what must be 30m of effort which may actually be pleasant?
5 pounds on the other hand is not only a small amount of money, but it’s also insulting to ask somebody that’s not a prostitute to sleep with one for such a pittance.
Fictional Churchill was acting like an asshole and the fictional socialite was acting rationally. She only should have replied instead “X million pounds is the best I can offer, but I should certainly hope you are good in bed Mr. Churchill”.
NB: I'm not in favour of this system - I'm only commenting on this one specific scenario.
So Apple claims your threat model is not technically possible.
Besides, Govt. can just order Apple to hand over the photos themselves from iCloud Photos because those are not end-to-end encrypted.
That's not at all the same as proactively casting a net and starting an investigation based on the results.
Apple has designed the system so that 30 matches are required; they could include more key material in each safety voucher to reduce the number required, or make it only require one match by providing the whole key in each voucher, or forego the system entirely in favor of one without such restrictions (which they can do, given some time, with an iOS update). It isn't "not technically possible" it's just "how they designed it", which is what the poster is saying Congress would ask about.
PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of.
If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’ve done something bad without evidence and imprison you. It happens all the time. It’s even happened in America. Just look up DNA exonerations - people have had DNA on the scene that literally proves their innocence and they’re still locked up.
People should care about their governments being corrupt, not necessarily about this.
Specificly the person doing the scanning already has access to photos and can double check the results without having to sieze the device, a rather public process.
Not necessarily. The FBI knocks on your door when they have convinced a warrant-signing judge, that’s there is probable-cause, or that additional information can be collected to build a case in which the defendant will ‘plead’ or make a deal for sentence reduction. 90% of defendants make a deal and never go to trial because by that time the stakes are so high a guilty verdict includes the harshest possible sentence.
FBI only needs to convince you they can win a case or the judge that there might be fruit behind a locked door, they don’t need direct evidence. It’s really up to the judge.
And we are not talking about some underpaid contractors making life altering decisions.
And we are also not talking about a system where just a false accusation can and will destroy one's life irreparably.
- You receive some naughty (legal!) images of a naked young adult while flirting online and save them to your camera roll.
- These images have been made to collide [1] with "well known" CSAM images obtained from the dark underbelly of the internet, on the assumption that their hashes will be contained in the encrypted database.
- Apple's manual review kicks in because you have enough such images to trigger the threshold.
- The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side.
- Your case is forwarded to the FBI, who now have cause to turn your life upside down.
This scenario seems entirely plausible to me, given the published information about the system and the ability to generate collisions that look like an arbitrary input image, which is clearly possible as demonstrated in the linked thread. The fact that most of us are unlikely to be targets of this kind of attack is little comfort to those that may be.
[1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
Given how most companies have barely paid, clueless and PTSD suffering human reviewers, a litany of mistakes is to be expected.
We should expect all the coherence of the twitter's nipple policy, except now it put you in jail or at least ruins your life with legal fees
Not to mention your scenario requires someone you trust trying to "get you." If that's true, then none of the other steps are necessary since you're already compromised.
And as far as your point about "someone you trust trying to 'get you'"... have you ever dated? Ever exchanged naughty photos with somebody (I expect this is even more popular these days among 20-somethings since covid prevented a lot of in-person hookups)? This doesn't seem crazy for a variant of catfishing. I could easily see 4chan posters doing this for fun.
The law states that if you find an image that's believed to be CSAM, you must report it to the authorities. If Apple's model detects CSAM on the device, sending the whole image to the moderation system for false positives carries the risk of breaking the law, because the images are likely going to be known CSAM, since that's what the database is intended to detect, so they'd be accused of storing it knowingly. Perhaps that's why the thumbnail system is needed.
So why wouldn't Apple store the files unencrypted and scan them when they arrive? That would mean Apple would remove themselves from liability by preventing themselves from gaining knowledge of which images are CSAM or not until they're scanned for, but could still send the original copy of the image with a far lower chance of false positives when something is found. That knowledge or the lack of it about the nature of the image is the crucial factor, and once they believe an image is CSAM they cannot ignore it or stop believing it's CSAM later.
That question may hold the answer to why Apple attempted to innovate in how it scans for child abuse material, perhaps to a fault.
A corrupt government has to know they want you "to just get you". Instead they will embed a collision in anti-government meme. That collision will flag you, and now they know you harbor doubts and will come get you.
This is why it's a privacy concern. It's no the tech (like you said photo dna's been about forver), it's the scanning of the phone.
Not without the risk of it being discovered (either through a leak or because someone analyzes the software on the phone), and then having a much bigger scandal on hand.
You don't even have to implement the feature. All you need is really good proof that you were asked to, and now your job at Apple is secure, along with a huge raise, for years if not for life. Something commensurate with what you and they both know they would lose in the PR damage and possible EU fines.
Until the EU makes it a legal requirement. Which they're getting close to
Also, they'd probably not use a Cupertino developer. I'm sure a dev in a nation with a lot less rights is easier for this sort of work. Find a nation where the protections for employees are worse and good jobs harder to find.
Photos people send me to my Android are automatically sent through 3rd parties, either through MMS, Facebook messenger, Google Photos, or One Drive. Photos arriving on my device are almost guaranteed to be uploaded to both OneDrive and Google Photos based on how defaults of Android phones are setup.
So someone could already send hash collisions my way (purposely or inadvertently) and authoritarian governments already have access in their respective clouds (at least China does).
And yet, there are not stories of people being falsely accused of child porn due to PhotoDNA hash collisions.
Why does "on device for apple devices only" change the calculus.
Everyone suspected they were scanning images for this and a number of other things on their services.
[1]: https://www.pcmag.com/news/fbi-sold-criminals-fake-encrypted...
After all, they want to suppress dissent, and they can't catch and arrest everyone - it's much more effective if people don't even dare to voice their dissent.
This is also why you see people in situations like the Arab Spring use known-insecure/monitored means of communication, because they realize that the value and power that comes from communicating and finding other likeminded people is worth painting a target on yourself (because you can't succeed without it, and if you succeed, there will be too many to prosecute).
Only if that government has worked out some deal with Apple whereby such an anti-government meme would end in the government being notified accordingly. Don't forget that you need a sufficiently high number of collisions, for one, and that those collisions are audited by Apple before being sent to law enforcement.
I can tell you from working in this area that it's possible for someone to have their lives ruined by a misplaced investigation, have that investigation abandoned because they turn out to be obviously innocent, and for that to not be well-known, because people simply would not understand the context.
Before this Apple scandal, if you'd written to your reepresentative or a journalist or an activist group and said "I was framed for child abuse because of computer program that misidentified innocent pictures", they would attach a very low priority to dealing with you or publicising this. And almost all people who have experienced this kind of nightmare really don't want to re-live it in public for some tiny possibility of real justice being served for them, or for others. They just want it to all go away.
Maybe it's because Google and Microsoft and others' policy of security through obscurity actually succeeded in preventing the details of PhotoDNA from coming to light, and it took Apple exposing their hashing model to reverse engineering by including it on the device for people to finally wake up.
- PhotoDNA
- CSAM scanning on cloud photo platforms
- the acronym "CSAM"
Before this whole Apple client-side scanning debacle... seems pretty likely. A lot of privacy-focused people also avoid Google and Microsoft cloud services like the plague and trusted Apple up to this point to protect their privacy. The fact that Apple was (and is) scanning iCloud Photos libraries for CSAM unbeknownst to most of us is just another violation of that trust and shows just how far the "what happens on your iphone, stays on your iphone" privacy marketing extends (read: not past your iphone, and sometimes not even on your iphone).
"U.S. law requires tech companies to flag cases of child sexual abuse to the authorities. Apple has historically flagged fewer cases than other companies. Last year, for instance, Apple reported 265 cases to the National Center for Missing & Exploited Children, while Facebook reported 20.3 million, according to the center’s statistics. That enormous gap is due in part to Apple’s decision not to scan for such material, citing the privacy of its users." [1]
[1] https://www.nytimes.com/2021/08/05/technology/apple-iphones-...
That, combined with a (pretty reasonable) expectation that the pool of sus hashes will greatly expand as law enforcement and others begin to understand what a convenient side-step this is for due process.
In addition, “pretty reasonable” is an opinion not fact. Where is the evidence that PhotoDNA hashes have been compromised in this way in the fifteen years they’ve been used?
First, how many people really understood this previously? Did society at large actually knowingly accept the current state of things, or did it just happen without most people realizing it? Even here on HN where we'd expect to find people way more knowledgeable about it than in general I'm not sure how well known it was about what was actually happening, though I'd assume most would be aware it was possible.
Secondly, there's a significant difference between your own device or Apple's server doing this. On the technical side of things, right now, it might not matter that much since it currently is limited to things you upload to iCloud. But more philosophically, it's your own device being turned against you to check you for criminal behavior. That's very different from somebody else checking up on you after you willingly interact with them.
This literally only works once you willing send photos to iCloud.
Wait, what?
https://ec.europa.eu/transport/themes/its/road/action_plan/e...
It's like announcing to your family member you're going to tell your neighbor you committed a crime and your family member turns you in first. Yeah, you could expect your neighbor to do the same, but are you really not going to feel any differently about the fact it was your family that turned you in?
All because you clicked on a link.
Phishing attacks can now put you in prison, label you as a pedophile and sex offender, and destroy your life.
a) That you know of b) Apple won't be doing this either
> Apple is building it.
No, they aren't.
Even Pegasus wasn't that much rotten, it at least wasn't indiscriminately installed onto everyone's phone.
But what can I say, there wasn't much uproar about piracy tax on blank CD-R(W) media back in the day, so why not have that now. And eventually we go peak USSR where everyone and their dog is suspect and whoever is arrested is the enemy of the people. Yay, it's somehow reassuring to know I won't live long enough to see it.
Is that really true? My understanding is that the manual reviewers at Apple only see some kind of low-resolution proxy, not the full-resolution image. I'd also be shocked if the human reviewers were shown the original, actually CP image, to compare to.
Given that, it's not necessary to produce an actual visual match, it's just necessary to produce an image that when scaled-down looks like CSAM (e.g. take an actual photo of a kid at the beach and photoshop in some skin-coloured swimwear with creases in the right places).
> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America?
The attack I'd worry about here is similar to swatting. Someone who doesn't like me sends a bunch of images like the ones I described above (not just one), they end up synced to iCloud (because Apple wants _everything_ synced to iCloud) and Apple reports me to the authorities, who end up knocking at my door and arresting me.
Even though I'm innocent, I'll probably have most of my computers confiscated for a while and spend a few days locked up.
> PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of.
PhotoDNA's algorithms and hashes aren't public, so it's not clear how an attacker would exploit PhotoDNA in the way that people are afraid will be done for Apple.
PhotoDNA also isn't, as far as I know, part of a product that aims to create unprotected backups of the phones of nearly a billion users. Apple really wants you to upload your whole phone to iCloud. The only comparable alternative is Google's Android backup but Google does the right thing and end-to-end encrypts that.
But a bigger concern is automatically. WhatsApp for example can automatically save all received photos to your Camera Roll, which is of course automatically backed up to iCloud for many people. So an attacker could potentially just send you a bundle of 40 images of however many and your phone automatically sends it to Apple.
"Following the 2004 Madrid train bombings, fingerprints on a bag containing detonating devices were found by Spanish authorities. The Spanish National Police shared the fingerprints with the FBI through Interpol. Twenty possible matches for one of the fingerprints were found in the FBI database and one of the possible matches was Brandon Mayfield. His prints were in the FBI database as they were taken as part of standard procedure when he joined the military."
"The FBI described the fingerprint match as '100% verified'."
Just reading those words is rage-inducing, but I'm grateful to have learnt this example of government lying. I feel like it should become an expression that societies teach to their children to warn them about abuses of power. Other mottoes synonymous with government deception and corruption come to mind, but at the risk of being too controversial I will share only their initials and dates: "SAARH" (2013), "MA" (2003), "IANAC" (1973), "NHDAEMZE" (1961), "AMF" (1940).
Nice big clean sample? It probably actually matches. Small sample? Mixed with other people's DNA? Especially in a place they tend to visit (or live in?)?
Sure, the probability is lower than getting struck by lightning. I certainly don't play in the rain and I won't be using cloud services where I'm exposed to this kind of nonsense with the FBI.
I don't think anyone can actually know that, because I don't think statistics are kept on how often these sort of dragnet programs result in civil liberty violations and secret grand juries. That should be the more immediate concern, because after that point you are depending on the goodwill of prosecutors... which is a super bad idea.
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
Threshold Secret Sharing Synthetic Match Vouchers
Not they haven't. The files are still uploaded to Apple's servers where Apple holds their encryption keys. Apple or the FBI could scan the photos for CSAM whenever they want to.
I believe a single image flag could be used as pretext to arrest someone that the government has already deemed an enemy of the state. Ex. Someone like Julian Assange
being on a government maintained secret list of CSAM flagged would probably have a lot of consequences in your life going forward without you knowing about that. It may be just one innocent image which accidentally matched the hash, yet without any procedure to get you off that list or even just to learn whether you're on that list ...
>If some corrupt government wants to get you they don’t need this.
it isn't corrupt government which is a threat here. It is a well intended well functioning relatively uncorrupt government which spends a lot of effort to protect its society from terrorism, child porn, sex trafficking, drugs, etc. In case of corrupt government the situation is kind of easier - you can always learn and correct necessary things through the corrupt government officials by paying a corresponding "fee".
https://en.wikipedia.org/wiki/Paul_Reubens#2002_pornography_...
-> if some corrupt government wants to *FIND YOU SHARING IDEAS THEY DON'T LIKE* they can *DEFINITELY* use this
I’m guessing most times this happens, the accused try to keep it on the DL.
https://www.wmur.com/article/fbi-raids-radio-talk-show-host-...
I understand that the system as stated today has multiple safeguards against such things happening but...
Given the sum-total of bs I've seen happening in this country, yes, believe that kind of thing is quite possible.
This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined.
What's even worse, when those individuals are facing whatever they're facing, they'll have to argue against the one-in-a-billion odds. If there are jurisdictions where defense lawyers don't get access to the images their client is accused of, and prosecutors and judges don't look at the images but only at the surrounding evidence (which says this person is guilty, with 99.9999999% certainty), Apple may have built a system that's statistically expected to send an innocent person to prison.
No one will have their lives ruined. If there's a false collision, someone at Apple has to look at the images as a final safeguard. If it's not actually CSAM then it won't ever make it to law enforcement.
1 in 10? 1 in 100? 1 in 1000? Pick a number, you now have an estimate of how much room for error there is in the whole system.
If you consider 0.15 lives ruined on average per year acceptable, and the reviewers have a 1-in-1000 error rate, then the rest of the system has to make less than 1-in-10-million mistakes per year. And I'm pretty sure 1-in-1000 for the reviewers is very, very optimistic, even if you do quality checks, control for fatigue, etc.
On the other hand, hopefully there are some safeguards after the reviewers (e.g. human prosecutors who don't blindly rubber stamp). But my point is: The room for error in anything done at scale that has severe consequences is extremely small.
And a country in which this isn't possible, is likely one that is going to ruin people's lives just fine without Apple.
A billion users with a thousand photos each would only be one trillion photos.
So a chance someone inadvertently has to look at a photo and … not prison, but, “nah, not this photo”.
Probability seems rather higher that the global suspicion-less and warrantless search will generate more positive PR results and systemic downstream negative privacy impacts than that chance of a subsequently easily avoided adverse result.
Odds are decent this all turns out seeming to normals like a “no downsides” implementation. If anyone wants to screech, better screech before it all proves perfectly splendid.
For example, a year ago I was trying to track down how some unscrupulous photos ended up in my photo library. I finally realized that Telegram had, without my knowing permission, been adding all photos I had received in a crypto discussion group to my Apple photo library.
There seem many ripe opportunities for targeting.
In my opinion that last step is not okay. I suppose the 30-image threshhold is a mitigating factor, really imo Apple is making their problem into my problem. I want to purchase from a company that offers me the peace of mind not to even have to think about such concerns, price isn't an obstacle. If Apple can't cater to my needs I hope another company will.
Since this only affects people using iCloud Photos, have you considered not doing that, or using an alternative that you believe isn't doing this?
For example, a neural net's cost function is just a multivariate function with weights as its input. To figure out how to move those weights (positively or negatively), the gradient of the function is calculated and the weights are nudged in the opposite direction (gradient is the direction of the largest growth of a function, we are trying to minimize the cost). Now, assume we are given a cost function and the weights are constant, now, the input can be the image. So, we take the gradient of the cost function with respect to the image pixels and can now see how we should nudge those to maximize the cost. Apple will absolutely need to protect against adversarial attacks for this to be viable. I'm hopeful.
The number of images in that database could well be far in excess of a million. According to NCMEC [1], in 2020 65.4 million files that were reported to them, and "[s]ince the program inception in 2002, CVIP [child victim identification project] has reviewed more than 330 million images and videos."
Of course many of those were duplicate but it would be entirely unsurprised if there were more than a million original files.
Just want to surface that because the title seems a little sensational based on the conclusion.
I think you could just do "sort | uniq -c | sort -nr" on the neuralhash values to find the most frequently occurring ones pretty fast?
I don't know what the author means by "taking advantage of the birthday paradox". If they're referring to the "birthday attack" [0], I don't think it makes sense. The birthday attack is a strategy that helps you find a collision without hashing every single image, but he states that he already generated NeuralHashes for all 1.43 million images.
Furthermore, isn't there a simple linear time algorithm to detect collisions given that you already have all the hashes? Iterate over your NeuralHashes and put them into a hash table where the NeuralHash is the key, and the number of occurrences is the value. Whenever you insert something into a bucket and there's already something there, you have a neural hash collision.
[0]: https://en.wikipedia.org/wiki/Birthday_problem#Probability_t...
Basically it could become an arms race where, say, free speech advocates convince people to just keep and share some images and overwhelm the process. Then Apple adapts to new hashes, blocklists known false positives, and the cycle repeats.
Someone got more details on that? How does the birthday paradox come into play here?
Edit: I'm procrastinating so I tried it. It's 8 lines including IO/parsing and runs in 2.8 seconds on my laptop:
import collections
hash_to_filenames = collections.defaultdict(list)
with open('hashes.txt') as f:
for line in f.readlines():
filename, hash = line.strip().split()
hash_to_filenames[hash].append(filename)
dupes = {h: fs for h, fs in hash_to_filenames.items() if len(fs) > 1}
print(f'Done. Found {len(dupes)} dupes.')
The algorithm sounds like a simple tree search algorithm. Let's consider the naive case: traverse all images, and keep a list of hashes you have already visited. For every extra image, you have to traverse all previous n hashes you have previously computed. Naively doing this check with a for loop would take O(n) time. You have to do this traversion for every image, therefore total time complexity is O(n^2).
Fortunately, there is a faster way to check whether you have found a hash before. Imagine sorting all the previous hashes and storing them in an ordered list. A smarter algorithm would check the middle of the list, and check whether this element is higher or lower than the target hash. When your own hash is higher than the middle hash, you know that if your hash is contained within the list, it is contained in the top half. In a single iteration you have halved the search space. By repeating this over and over you can figure out if your item is contained within this list in just log_2(n) steps. This is called binary search. Some of the details are more intricate (e.g. Red-Black trees [1], where you can skip the whole sorting step) but this is the gist of it.
This all sounds way more complicated than it is in practice. In practice you would simply `include <set>;` and all the tree calculations are done behind the scenes. The algorithm contained within the library is clever, but the program written by the author is probably <10 lines of code.
They mean Russia and Belarus would need to agree to add a file to the list? Yeah, this is a very hard barrier to overcome! /s
This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark.
I do not presume such a finding of photos is easy to come by but I wish the author put details on the sample set.
I'd love to see this work extended; if you find additional collisions in the wild please submit a PR to the repo (please do not submit artificially generated adversarial images): https://github.com/roboflow-ai/neuralhash-collisions
For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes)
[1] https://tidbits.com/2021/08/13/new-csam-detection-details-em...
> In Apple’s tests against 100 million non-CSAM images, it encountered 3 false positives when compared against NCMEC’s database. In a separate test of 500,000 adult pornography images matched against NCMEC’s database, it found no false positives.
For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes)
Thank you for those helpful tidbits.
You'd expect the same result even if the incidence in porn was 10x higher than in typical images (30 in 100 million = 0.15 in 0.5 million).
Though knowing that 1 in roughly 30 million images generates a false positive is the most important figure I suppose. Assuming 100 million iPhones with each 1000 pictures that would generate some 3000 phones with one or more false positives [1] and a roughly 5% chance that some phone has at least 2 false positives.
[1]: https://www.wolframalpha.com/input/?i=%28100+million%29+e%5E... [2]: https://www.wolframalpha.com/input/?i=+%281+-+e%5E-l+%28e%5E...
If you can make an innocent picture collide with a CSAM picture, presumably you can also edit a CSAM picture to have a random hash not in the database?
* edit - this was 25 years ago, admittedly, and much of what I downloaded came from randos on AOL. Things might be different now.
The point of the article is that the algorithm that Apple is using works as well as Apple says it does, and the headline implies that it doesn't.
I think the headline should be:
ImageNet contains about as many naturally occurring NeuralHash collisions as Apple predicts
I just read today that OnlyFans also contribute hashes to the same organisation, I wonder how many other companies have been doing the same for years.
Haven't see any uproar about this elsewhere for other services. For example, CloudFare has had this as an option on their environment for years.
This was discussed on HN yesterday here (and a few other front-page stories): https://news.ycombinator.com/item?id=28218391
> You will need 4 files from a recent macOS or iOS build:
neuralhash_128x96_seed1.dat
NeuralHashv3b-current.espresso.net
NeuralHashv3b-current.espresso.shape
NeuralHashv3b-current.espresso.weights
Not according to Apple. They're publicly claiming a one-in-a-trillion false positive rate from the automated matching. Either that's blatant false advertising or they're putting known (as in: more likely than not) CSAM in front of their human reviewers. Can't have it both ways.
> Further, Apple only gets a low-resolution version of the image.
Which makes zero difference with regard to the content being illegal. Do you think they would overlook you possessing an equally low-resolution version of the same photo?
> In any case, presumably such issues have been addressed, as neither the FBI nor NCMEC have raised a stink about it.
Selective enforcement; what else is new? It's still a huge risk for Apple to take when the ethically superior (and cheaper and simpler) solution would be to encrypt the files on the customer's device, not scanning them first, and store the backups with proper E2E encryption such that Apple has no access to or knowledge of any of the content.
Obviously you could not make the same argument yourself unless you were also a reputable megacorp. There are upsides to being king. In this case, NCMEC wants the actual perps in jail so they're not going to take shots at Apple or its employees on technicalities.
The FBI was given clearance to take over a website serving CSAM in order to catch more users of the site. As such, the FBI has technically distributed CSAM in the past.
https://www.dallasnews.com/news/crime/2017/01/17/the-fbi-ran...
If a bad actor can send targets innocent images which hash to CSAM, you essentially have an upgraded secret swatting mechanism that people will use to abuse others.
- Those images would either have to be actual CSAM (illegal to distribute in the first place and generally hard to get)
- fake collision images are placed instead, and all that would do is push the multiple fake CSAM images to Apple's human reviewers who would then likely mark it as a false positive/spam. If the Apple reviewer makes a mistake and does report it as CSAM them the only risk (in the US) is law enforcement having probably cause for search & seizure to look for other CSAM.
n(log n)? I thought the birtday paradox lets you expect collisions after sqrt(n) tries…
For example, my understanding of Microsoft’s PhotoDNA is that their perceptual hash has been reverse-engineered and that one could go backwards from a hash to a blurry image. But also it is very hard to get the list of PhotoDNA hashes for the NCMEC database. In other words, are Apple unintentionally releasing enough information to reconstruct a bunch of blurry CSAM?
Think more like a “deep dream” than these adversarial attacks.
They absolutely do not. They use Private Set Intersection to achieve that.
If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Is anyone so naive that they believe the CCP wouldn't deliver such an ultimatum? Apple's position seems to completely ignore recent Chinese history.
If Apple could stand up to them before this system, why can't they stand up to them with this system?
Now? Well now it's real simple. It will even conveniently not expose the actual images it's checking for. Apple now has significantly less ability to rationally reject the request based on the effort and difficulty of the matter.
Even Apple's own reasons to reject the request have decreased. It would have legitimately cost them more to fulfill this request before, even if China did want to play hardball. Now, they have greater incentive to go along with it.
In my mind there are two possible ways to view this.
We could trust Apple last month and we can trust them today.
We couldn't trust Apple last month and we can't trust them today.
I don't understand the mindset that we could trust Apple last month and we can't trust them today.
But isn't that exactly why this is such a big deal? It sets a precedent that it's ok that devices are scanning your local device for digital contraband. Sure, right now it's only for photos that are going to be uploaded to iCloud anyway. But how long before it scans everything, and there's no way to opt out?
I don't see this as so much a question of apple's trustworthiness, I see it as a major milestone in the losing battle for digital privacy. I don't think it will be long before these systems go from "protecting children from abuse" total government surveillance, and it's particularly egregious that it's being done by apple, given their previous "commitment to privacy".
Do we think this is detectable? If yes, then why worry about it if we will know when this switch is made? If not, why did we trust Apple that this wasn't happening already?
That is the primary thing I don't understand, this fear rests on an assumption that Apple is a combination of both honest and corrupted. If they are honest, we have no reason to distrust what they are saying about how this system functions or will function in the future. If they were corrupted, why tell us about this at all?
In this instance the specific question is "Do I trust Apple to be honest about when they scan our files?". I don't know why this news would change the answer to that question.
Did we do this already? If not, why did we assume this didn't exist previously?
>Is the inevitable Samsung version of this system going to be as privacy-preserving?
I don't see how this is Apple's fault. Be angry at Samsung for their system. I don't blame Snap if I have a problem with Instagram Stories.
>How are we sure the hash list isn't tainted?
How did we know the hash list wasn't tainted when doing this comparison on Apple's servers?
>All of these problems are solved by one principle: Don't put the backdoor code in the OS to begin with.
Apple controls the hardware, software, and cloud services. The backdoor didn't need to exist in the OS for there to be a backdoor.
It's amazing that Apple is still thinking about moving this idea forward, but they must think they are untouchable and can do what they want.
And it adds insult to injury that two very different images can generate the same NeuralHash value.
>If not, why did we trust Apple that this wasn't happening already?
I do not trust Apple. I don't really trust any major tech company, because they put profit first, and everything else comes second. I believe that a company as large as apple is already colluding with government(s) to surveil people, because if a money-making organization is offered a government contract that involves handing over already collected data, and it was kept secret by everyone involved, why would they refuse? I know that's very cynical, but I can't see it any other way.
But that's beside the point, which is that what apple is doing is paving the way for normalization of mass government surveillance on devices that we're supposed to own.
>If they were corrupted, why tell us about this at all?
So that we can all get used to it, and not make a big fuss when google announces android will do the same thing. It's much easier to do things without needing to keep them a secret. This is in no way only about apple, they're just breaking the ice so to speak.
Then you should have never been using a closed system like Apple in which they had control over every aspect of it. That is my fundamental point. I'm not saying you should trust Apple. I am saying this shouldn't have changed your opinion on Apple.
>So that we can all get used to it, and not make a big fuss when google announces android will do the same thing. It's much easier to do things without needing to keep them a secret. This is in no way only about apple, they're just breaking the ice so to speak.
I just need more evidence before I believe a global conspiracy that requires the coordination between both adversarial governments and direct business competitors.
That's the point. Yesterday someone posted a fantastic TSA metaphor where they are doing the same scans and patdowns but with agents permanently stationed in the privacy of your home where they pinkie promise it will only be before a trip to the airport and only checking the bags you will be flying with.
You know food poisoning is dangerous and you'll be safer with a food taster to make sure nothing you eat is spoiled. I'll just help myself to your domicile and eat your food to make sure it's all safe. I already made a copy of your keys to let myself in. It's for your own good.
If Apple can be cowed by China into adding fake CSAM hashes by threat of banning iPhone sales, they could be cowed to surveil Chinese citizens in the search for subversive material. It's no skin off China's back if it's harder for Apple -- they'll either make the demand or they won't. This changes basically nothing.
It's kinda true, but ignores how humans really work. Apple will be pushed around to a degree, but there will be limits. The harder the ask now the less China can ask later. And the more Apple can protest about the difficulty and impossibility and other consequences they will face, the more likely China is to back off.
Both sides want to have their cake and eat it too, and will compromise to make it basically work. But if China makes demands so excessive they get Apple to cut ties, China loses. Apple has the money, demand, customer loyalty, and clout to make things real uncomfortable. Apple would have to pay a hefty price, but if any company can do it... it's them.
So I don't think it's fair to say that no matter what China will just demand whatever whims strike it each day and everybody will play ball or gtfo. That just isn't how shit works.
The direct job losses are one obvious problem for the CCP but a company like Apple saying "We're moving production to Taiwan/Vietnam/US because of security risks in China" would be catastrophic for the (tech) manufacturing industry as a whole in China. No sane Western based CEO will want to be seen taking that security gamble.
Do I think Apple would do that and forgo the massive Chinese smartphone market? That's another story.
Apple has to store data of chinese citizens in china? And Apple has to adhere to chinese laws in china? How insane.
It's crazy how deluded the "CCP crowd" are. Apparently, the "CCP crowd" thinks companies are allowed to do business in another country and not abide by their laws.
Are you going to go insane since the EU requires tech companies to store EU citizens data within the EU?
https://blogs.microsoft.com/eupolicy/2021/05/06/eu-data-boun...
You would think most countries would demand their citizens data be stored locally.
https://en.wikipedia.org/wiki/1989_Tiananmen_Square_protests
https://en.wikipedia.org/wiki/Censorship_in_China
https://en.wikipedia.org/wiki/Overseas_censorship_of_Chinese...
https://en.wikipedia.org/wiki/Internet_censorship_in_the_Peo...
Google for "Operation Aurora"
Also see: https://www.reuters.com/article/us-china-apple-icloud-insigh... for a very similar situation to the one described up-thread.
I would be intrigued to hear about the Uyghurs thoughts on the matter, as well as those of Assange.
Separately, the US has even greater pressure on Apple in the case they want to unilaterally add database images, considering they have a actual chance and means to jail (and run through the legal ringer) whomever tells them ‘no’. And that’s just the overt pressure available; I think this is a more likely potential for trust violation here, even if both could come to pass.
EU, China, Iran, US: everyone gets to spy on their own children and forbid whatever they want.
On the flip side, though, at least Apple will have given us a canary. And that's why I don't think Apple will be asked to add these hashes: if the governments don't want their citizens to know what's being scanned server side, pushing the equivalent data to clients would tip their hand. They might just write Apple off as a loss and rely on Google, Facebook, etc.
It would be trivial to have the same payload on each device, and extract different answers using the matching server side db which varies by country.
Perhaps not trivial, but just short.
I honestly don't think Apple wants to deal with any of that crap and that they would rather silently can the system and do exactly what everybody else does than place themselves in the line of fire when their own unique trademark system is being abused.
Suffice to say, unless you provide proof, I am reasonably confident there’s no way to verify the hash db doesn’t contain extra hashes other than the CSAM hashes provided by the US government. But I’ve been wrong many times before.
If what you mean by blinded is that you don't know what the source image is for the hash, that's true. Otherwise Apple would just be putting a database of child porn on everyone's phones. You gotta find some kind of balance here.
What do you mean you can't verify it doesn't contain extra hashes? Meaning that Apple will say here are the hashes in your phone, but secretly will have extra hashes they're not telling you about? Not only is this the kind of thing that security researchers will quickly find, you're assuming a very sinister set of features from Apple that they'll only tell you half the story. If that were the case, then why offer the hashes at all? It's an extremely cynical take.
The reality is all of the complaints about this system went from this specific implementation, and then as details get revealed, it's now all about the future hypothetical situations. I'm personally concerned about future regulations, but those regulations could/would exist independently of this specific system. Further, Dropbox, Facebook, Microsoft, Google, etc all have user data unencrypted on their servers and are also just as vulnerable to said legislation. If the argument is this is searching your device, well the current implementation is its only searching what would be uploaded to a server instead. If you suggest that could change to anything on your device due to legislation, wouldn't that happen anyway? And then what is Google going to do... not follow the same laws? Both companies would have to implement new architectures and systems for complying.
I'm generally concerned about the future of privacy, but I think people (including myself initially) have gone too far in losing their minds.
At no point is anyone besides Apple able to view any NeuralHash hashes from the CSAM database. You can verify the database is the same on all iPhones, but you are not able to look at any of the hashes.
Is the issue that you want to take a given photo that you believe the CCP or whomever is scanning for, compute a NeuralHash, and then see if that's in the db? Or are you wanting to see if your db is different from other phone's db's? Because I think the later is the one that most people are concerned about.
You'd have to switch to a dumb phone. Assuming you can find one that works on contemporary networks.
didn't they already say the DB would be nation dependent?
> • Database update transparency: it must not be possible to surreptitiously change the encrypted CSAM database that’s used by the process.
> • Database and software universality: it must not be possible to target specific accounts with a different encrypted CSAM database, or with different software performing the blinded matching.
I mean, you can argue that Apple's safeguards are insufficient etc., but at least acknowledge that Apple has thought about this, outlined some solutions, and considers it a manageable threat.
ETA:
> Since no remote updates of the database are possible, and since Apple distributes the same signed operating system image to all users worldwide, it is not possible – inadvertently or through coercion – for Apple to provide targeted users with a different CSAM database. This meets our database update transparency and database universality requirements.
> Apple will publish a Knowledge Base article containing a root hash of the encrypted CSAM hash database included with each version of every Apple operating system that supports the feature. Additionally, users will be able to inspect the root hash of the encrypted database present on their device, and compare it to the expected root hash in the Knowledge Base article. That the calculation of the root hash shown to the user in Settings is accurate is subject to code inspection by security researchers like all other iOS device-side security claims.
> This approach enables third-party technical audits: an auditor can confirm that for any given root hash of the encrypted CSAM database in the Knowledge Base article or on a device, the database was generated only from an intersection of hashes from participating child safety organizations, with no additions, removals, or changes. Facilitating the audit does not require the child safety organization to provide any sensitive information like raw hashes or the source images used to generate the hashes – they must provide only a non-sensitive attestation of the full database that they sent to Apple.
[1] https://www.apple.com/child-safety/pdf/Security_Threat_Model...
Apple will cave. They've demonstrated the technology works. The CCP is paying attention. Look for this feature to be mandated across all phones soon. Pandora's box has been opened.
[0]: https://xkcd.com/538/ (this link comes up with terrifying frequency on HN)
This may have shifted over time as China, Russia, NK, Iran increase their attacks, but it doesn’t diminish the fact that the EU is also hacking the US without repercussions.
Great (depressing) Twitter account: Is EU Concerned? @ISEUConcerned Very, deeply, strongly, seriously, gravely, extremely, unprecedentedly
If we lived in a world where the people who make these kind of decisions for companies were actually accountable in this way, life might be better in a lot of different ways. But sadly we do not.
I thought countries often have under the table agreements with one another to explicitly spy on each others citizens, since its illegal for the country to spy on its own citizens. It's illegal for the other country too, but it's a lot easier to turn a blind eye to it.
These "everyone is doing it" statements are nonsense. Not everything is doing it.
The USA is probably doing it for them.
NSA tapped German Chancellery for decades, WikiLeaks claims
NSA spying row: Denmark accused of helping US spy on European officials
US caught spying on EU ‘allies’ AGAIN…not like the Europeans will do anything about it
?
[1]: https://tidbits.com/2021/08/13/new-csam-detection-details-em...
[1] I'm stealing the expression from this excellent article: https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
I believe Apple's intention is to accept hashes from all governments, not just one US organization. One of their ineffectual concessions to the criticism was to require two governments provide the same hash before they'd start using it.
And maybe even identify avenues for sharing that they haven't already identified and monitored/controlled (e.g. some encrypted chat app they haven't blocked yet).
Can you give some examples of this on the iPhone?
Also, it seems like on the spying front [1] (at least publicly, with Apple), they've preferred more "backdoor" ways to get access over more overt ones, so this scanning feature might encourage asks that they wouldn't have otherwise made.
[1] this contrasts with the censorship front, where they've been very overt
Don't like it? Don't sell phones. Apple will cave.
Note that you don't even have to arrest everyone. The fear is enough to prevent thoughtcrime.
Uh...this already happened. [0][1]
[0] https://www.macrumors.com/2021/05/17/apple-security-compromi...
"ostensibly" is the key word there. If the datacenter is physically located in China, then there's a CCP official on the board of the company that controls it.
Because Apple has already built that functionality, and it exists? What alternative dragnet currently exists to identify iOS users who possess certain images? This would be code reuse.
In both cases people are stretching to come up with hypothetical scenarios about how these systems could be abused by a government ("they could force Apple to insert non-CSAM hashes into their database" or "they could force Google to insert a backdoor into your app") while completely ignoring the elephant in the room: if a government wanted to do these things, they already have the power to do so.
If your concern is that a government might force Apple or Google to do X or pull product sales in their country, whether Apple performs on-device CSAM scanning vs scanning it on their servers, or whether Google signs your app vs you signing it doesn't materially change anything about that concern.
The outrage around this particular situation is even more confusing to me because you can opt out entirely by disabling iCloud Photos, and if you were already using iCloud Photos then the scanning was already happening on Apple's servers anyway, so the only actual change is that the scan now occurs before instead of after the upload.
VOGON GUARD: I’ll mention what you said to my aunt.
[Airlock door closes and locks]
FORD: Potentially bright lad, I thought.
ARTHUR: We’re trapped now, aren’t we?
FORD: Er… Yes, we’re trapped.
ARTHUR: Well didn’t you think of anything?
FORD: Oh Yes.
ARTHUR: Yes?
FORD: But, unfortunately, it rather involved being on the other side of the airtight hatchway they’ve just sealed behind us.
The way it would work is the government in question would submit legitimate CSAM but modified to produce a collision with a government target image. Looking at the raw image (or a derivative) a reviewer at Apple or ICMEC would see a CSAM image. The algorithm would see the anti-government image. So Apple scans Chinese (or whoever) citizens libraries, finds "CSAM" and reports them to ICMEC which then reports them to the government in question.
Every repressive government and some notionally liberal governments will eventually do this. It likely is already happening with existing PhotoDNA systems. The difference is that's being used by explicit sharing services where Apple's new system will search for any photo in a user's library regardless of it being "shared" explicitly.
If Apple finds that a particular hash is notorious for false positives, they can reject it / ask for a better one. And they’re not scanning your library; it’s a filter on upload to iCloud. The FUD surrounding this is getting ridiculous.
We’re talking about a company that makes phones and computers, and sells music and tv shows via the internet. Does that matter at all?
How about this. All car manufacturers must now wirelessly transmit when the driver of the car is speeding immediately. How about that?
Let’s just go all out and embed law enforcement into all private companies.
This is fascism, the merging of corporations and the government.
Apple complied.
https://indianexpress.com/article/technology/mobile-tabs/app...
I just feel so lost and powerless these days, I don't know how much longer I can go on when every piece of technology I own is working against me - tools designed to serve a ruling class instead of the consumer. I don't like it one bit.
Convenience and fashion tend to trump security and principles for most people. (Oftentimes, I'm one of those people as well, though I try not to be. It's exhausting to be an activist 100% of the time. But let's keep at it!)
For additional fun, strike a deal with the #2 non government owned carrier in whichever country you do this to. Offer the iPhone at a special rate for a few months. They would kill the government telco while selling record numbers of phones with free publicity. And at the same time scare any other government into not trying this kind of stunt with Apple ever again.
Then there's China; 17% of Apple's global revenue, $43.7bn. I don't think shareholders would much appreciate that.
> the public in India would either put pressure on their government or buy iPhones via import websites
The iPhone had 2.97% market share in India in April 2021, down from a high of 3.54% in June 2020. I don't think the people who wanted to buy iPhones but couldn't would be able toput any significant amount of political pressure on politicians.
Rich people would just import them from somewhere else like they always have before, and everyone else would switch to some available Android phone that had the modifications that the government wanted.
That's not at all what the lefty geeks learned by reading Chomsky or what the righty geeks learned by reading Heinlein.
All along these people thought algorithms and protocols (e.g. bitcoin and TCP/IP) would somehow be a powerful force that would cause governments to fall on their knees and let people evade government control. After all, it's distributed! You can't stop it!
Well, that was all very foolish, because they mistook government uninterest in something for the equivalent of government being powerless to control it, and when governments did start taking an interest in something, it turns out that protocols and algorithms are no defense against the realities of political power. It is to the field of politics, and not the field of technology, that one must turn in order to increase collective freedoms. Individual freedom can be increased by obtaining money or making lots of friends, but collective freedom cannot be increased this way, it can only be increased by organizing and influencing government.
Remember the people who, after the invention of the phone, insisted that it was a nice trick but probably only useful for a few businessmen with dictation needs?
Yeah, they all had to change their tone at some point, under the shame of having been wrong for so long.
Not sure everyone has gotten the memo yet.
But that's the underlying problem here. Apple isn't a standardized protocol or a distributed system. It's a monolithic chokepoint.
You can't do this with a PC. Dell and HP don't retain the ability to push software to hardware they don't own after they've already sold it and against the will of the person who does own it.
People pointed out that this would happen. Now it's happening. Qué sorpresa.
Hell, Dell has shipped more than one bug that allowed attackers administrator-level access or worse, I wouldn't put it past them to come up with some kind of asinine feature that not only lets them push new software/drivers/whatever to the machine, but lets attackers do so as well.
If they did that, people would wipe their device and not reinstall the Dell software. Many people do this already as soon as they buy the device.
The internet and its design and associated protocols were designed to work around external forces - a nuclear attack or natural disaster. It was never designed to be government-proof. People who thought that would be the case were being idealistic and naive.
If you want real change in the world, as you said, you have to affect the political world, which is an option available to any citizen or corporation who can spend millions on lobbyists.
This made me think of the Butlerian Jihad in Dune:
"Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."
I'd say the typical remedy that societies have adopted for these sorts of things is legislation, though regulatory capture[1] is an issue that blocks the way.
This makes me very sad.
Correct me if I'm wrong but this feature needs an app install from the app store.
It's a feature that's could be done by anyone using sms anyway.
I guess apple had to add an easy way
Why doesn't the US govt follow India's govt? I've read that Americans can receive up to 4 unsolicited calls a day.
When the hell did it become Apple’s job to do this? Apple is not a branch of law enforcement. The government needs warrants for stuff like this. We are merging corporate and government interests here. Repeat after me, Apple is not supposed to be a branch of law enforcement.
It also says a lot about us, that we are beholden to a product. We have to ditch these products.
Apple provided a pathway, however unintentionally, to greater power. And those in power used their existing authority to gather even more for themselves, as they always do.
Like drops flow into streams into rivers into oceans, power aggregates at the top until regime change spills it back to the ground.
IMHO this is great, I wish more countries would enable this feature. Something like 95% of my phone calls are spam, to the point where I just don't answer the phone anymore unless they're in my contacts list. Users being able to actually report them as spam might actually result in this BS finally stopping.
> If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Is anyone so naive that they believe the CCP wouldn't deliver such an ultimatum? Apple's position seems to completely ignore recent Chinese history.
Apple policy << Local laws and regulations. It's very hard to believe their policy is anything less than a deliberate PR smokescreen meant to disarm critics, because it has so many holes.
Edit: just thought of another way Apple's policy could be easily circumvented and therefore cannot be regarded as a serious proposal: get two countries to collaborate to add politically sensitive hashes to the list (e.g. China and North Korea, or China and Cambodia). That doesn't even require Apple to be coerced.
Then you could provide a proof that an image which is the same as the "censored", one except for the masked part, has the perceptual hash specified. I don't know if this is technically feasible (but I'd be happy for someone knowledgeable to opine). I also admit that there are secondary concerns, like the possibility of recognising the background image, and this being used by someone to identify the location, or tipping off a criminal.
Probably it would only be appropriate to do this in the case of someone being accused, and maybe then in a way where they couldn't relay the information, since apparently they don't want to make the hash database public.
Also, for the record, I'm spitballing here about infosec. This isn't me volunteering to draw black boxes or be called by anyone's defense.
If they add some new hashes I presume that would be harder to spot and isn’t going to be advertised
A few bytes changed in a binary blob that would enable a backdoor would be almost impossible to detect.
Nobody really knows what the blobs do. They likely have paved the way for Stingrays and other devices.
If Apple really wanted to they could secure the iPhone. Just offer a 2x higher bug bounty than any government and things like stingray would not work.
Although I think recent iphone designs don’t give the modem DMA so it’s still better than Android but I wouldn’t consider it secure.
As much as Apple wants access to the Chinese market, it would (presumably) draw a line at some point where it would (presumably) have to choose between that market and the US market, if only because the latter is both its legal domicile and the source of most of its talent.
Version A: CCP wants to exploit the hash database, there are lots of ways to do that, bullying Apple is one, any other way gives Apple a "we are looking into it" excuse. "We must comply with local laws, but we will not change our software bla bla."
Version B: CCP wants to exploit iOS, only way to do it is to bully Apple, this forces Apple's hand and very possibly Apple moves production (not just sales) out of China because they no longer trust they will be offered "plausible deniability."
I'm sure there are lots of reasons for that absurd cash reserve, but my best guess is it's to cover the eventuality of B. above; Apple talking about that publicly would be tricky.
https://www.cnbc.com/2020/07/30/apple-q3-cash-hoard-heres-ho...
As a matter of fact, I am not sure that would be possible: It might well be that no other country has the capacity (machines and labour) to churn out that many iPhones. Would be interesting to hear if anyone has insight on that. (Tim Cook presumably knows...)
I admit it's total speculation, but I think the massive cash reserves are for that: to weather a disruption in production facilities and move production to a more US-friendly location.
It seems like it wouldn't take that. If you can generate a colliding pair of images, you could probably create a pair where one of the images might get attention with child porn groups and thus, shared around enough to end up in the CSAM database. And where the other was innocuous.
Or maybe China already said "put in this CSAM check or you can't make or sell phones in China".
Since Apple's position is contrary to their previous privacy policy and doesn't seem to make a lot of sense, it's quite possible extortion already happened (and not necessarily by China).
Hey Apple, we can either turn up the anti-trust heat by a lot, or we can turn it down, which is it going to be? Except it would be couched in intellectually dishonest language meant to preserve a veneer that the US Government isn't a violent, quasi-psychotic bully ready to bash your face in if you don't do what they're asking.
The interactions with intelligence about the new program would begin after they acquiesce to going along.
It's too easy. There's an extraordinary amount of wilful naivety in the US about the nature of the government and its frequent power abuses (what it's willing to do), despite the rather comically massive demonstration of said abuses spanning the entire post WW2 era. Every time it happens the wilfully naive crowd feigns surprise.
I think that the real risk is, "put these hashes in your database or we will stop all iPhone related manufacturing in China".
Unless the goal is to simply the effort/expense of scanning by making it a client process.
And there’s no way the CCP would order Apple to do that.
At best, they would look biased. At worst, they would be sending a signal that they don’t care about Chinese children.
You wouldn’t even need government strong arming, mobs of netizens would happily tear Apple down.
They'd do it directly because it's expedient and useful. If you're operating such a sprawling authoritarian regime, it's important to occasionally make a show of your power and control, lest anyone forget. The CCP isn't afraid of Apple, Apple is afraid of the CCP. Lately the CCP has been on a rather showy demonstration of its total control. If you're them it's useful to remind Apple from time to time that they're basically a guest in China and can be removed at any time. You don't want them to forget, you want to be confrontational with Apple at times, you want to see their acknowledged subservience; you're not looking to avoid that power confrontation, the confrontation is part of the point.
And the threat generally isn't made, it's understood. The CCP doesn't have to threaten in most cases, Apple will understand ahead of time. What gets made initially is a dictate (do this), not the threat. If something unusual happens, such as with Didi's listing on the NYSE against Beijing's wishes (whereas ByteDance did the opposite and kowtowed, pulling their plans to IPO), then, given that Didi obviously understood the confrontation risk ahead of time and tested you anyway, then you punish them. If that still isn't enough, you take them apart (or in the case of Apple, remove them from the country).
To also clarify, the avenue of extortion isn't open to every country. But the avenue I presented is as long as that country has an ally. I'm not aware of any country not having an ally, so I presume that this avenue is pretty much open to any country.
This policy really needs to be compared to the status quo (unencrypted on cloud image scanning). When you compare in transit client side hash checks that allow for on cloud encryption and only occur when using the cloud it's hard to see an argument against it that makes sense given the current setup.
The abuse risk is no higher than the status quo and enabling encryption is a net win.
I would think they could more easily just make your phone carrier install a malware update on your phone, rather than jumping through all of these hoops to get them access they already have.
Plenty of data is leaking out of people's phones already as can be seen from, e.g. the Parler hack.
Now, what has actually changed? The two compelling push-back against a country's demands for more surveillance etc. are:
a) it is not technically feasible (eg, wiretapping E2EE chats), and
b) it is making the device insecure vis-a-vis hackers and criminals (eg, putting in a backdoor)
The interesting question is: Have these defences against government demands been weakened by this new technology? Maybe they have, that would be my gut feeling. But it is not enough to assert it, one must show it.
1. USA
2. UK, Canada, Australia, New Zealand
Now that we know that collisions are not just easy -- but happen by themselves with no human input (as evidenced by the Imagenet collisions), we know this system can't work. Apple has two remaining safeguards: a second set of hashes (they say), and human reviewers.
The human reviewers are likely trained to prefer false-positives when unsure, and so while a thorough human review would clearly indicate "not CSAM" for the images the malicious collisions match, it doesn't feel like much of a safeguard to me. (Remember the leaked memo from the US's organization -- they called our objections "screeching voices". I'm sure the actual reviewers think similarly.)
I assume the people reviewing CSAM for the CCP will be in China, so they can be in on the whole scheme. (In the US, we have slightly better checks and balances. Eventually the image will be in front of a court and a jury of your peers, and it's not illegal to have photos that embarrass the CCP, so you'll personally be fine modulo the stress of a criminal investigation. But that dissident in China, probably not going to get a fair trial -- despite the image not being CSAM, it's still illegal. And Apple handed it directly to the government for you.)
I don't know, I just find this thing exceedingly worrying. When faced with a government-level adversary, this doesn't sound like a very good system. I think if we're okay with this, we might as well mandate CCTV in everyone's home, so we can catch real child abusers in the act.
They do say any press is good press, though.
>Apple's NeuralHash perceptual hash function performs its job better than I expected
When you are just looking for any two collisions in 100M x 100M comparisons, of course you'll find a small number of positives, Apple said as much. The number of expected collisions will scale linearly with the number of 'bad' images, which is not 100M. Assuming it's 100k, we'd expect 1000x fewer collisions in ImageNet, or ~0.002 collisions, which is effectively 0. It's the artificial images that will potentially sink all this, not a very low rate of naturally occurring hash collisions.