Convert Apple NeuralHash model for CSAM Detection to ONNX
github.com
github.com
Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’ - https://news.ycombinator.com/item?id=28225706
Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068
This is a little unexpected. I'm not sure whether this has any implication on CSAM detection as whole. Wouldn't this require Apple to add multiple versions of NeuralHash of the same image (one for each platform/hardware) into the database to counter this issue? If that is case, doesn't this in turn weak the threshold of the detection as the same image maybe match multiple times in different devices?
This is by-design - The whole idea of a perceptual hash is that the more similar the two hashes are, the more similar the two images are, so I don't think it invalidates any claims.
Perceptual hashes are different to a cryptographic hash, where any change in the message would completely change the hash.
This is already proven to be inaccurate. There are adversarial hashes and collisions possible in the system. You don’t have to be very skeptically-minded to think that this is intentional. Links to examples of this already posted in this thread.
You are banking on an ideal scenario of this technology not the reality.
EDIT: Proof on the front page on HN right now https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
If that is the case, then the word "hash" is terribly mis-applied here.
Not if their processor architectures are all the same, or close enough that they can write (and have written) an emulation layer to get bit-identical behaviour.
My first thought was that they didn't want to make the model too easily accessible by putting it on macOS, in order to avoid adversarial attacks.
But knowing this now, Intel Macs are an issue as (not as I previously wrote because they differ in floating point implementation to ARM, thanks my123 for the correction) they will have to run the network on a wide variety of GPUs (at the very least multiple AMD archs and Intel's iGPU), so maybe that also factored in their decision ? They would have had to deploy multiple models and (I believe, unless they could make the models exactly converge ?) multiple distinct database server side to check back.
To people knowledgeable on the topic, would having two versions of the models increase the attack surface ?
Edit: Also, I didn't realise that because of how perceptual hashes worked, they would need to have their own threshold to matching, independent of the "30 pictures matched to launch a human review". Apple's communication push implied exact matches. I'm not sure they used the right tool here (putting aside the fact for now that this is running client side).
Edit: cwizou correctly points out not all of the features (per Apple) will be on Monterey but the code exists.
- Communication safety in Messages
> "This feature is coming in an update later this year to accounts set up as families in iCloud for iOS 15, iPadOS 15, and macOS Monterey."
- CSAM detection
> "To help address this, new technology in iOS and iPadOS"
- Expanding guidance in Siri and Search
> "These updates to Siri and Search are coming later this year in an update to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey."
So while the two other features are coming, the CSAM detection is singled out as not coming to macOS.
But ! At the same time, and I saw that after the editing window closed, the GitHub repo clearly states that you can get the models from macOS builds 11.4 onwards :
> If you have a recent version of macOS (11.4+) or jailbroken iOS (14.7+) installed, simply grab these files from /System/Library/Frameworks/Vision.framework/Resources/ (on macOS) or /System/Library/Frameworks/Vision.framework/ (on iOS).
So my best guess is, they trialed it on macOS as they did in iOS (and put the model there contrary to what I had assumed) but choose not to enable it yet, perhaps because of the rounding error issue, or something else.
Edit : This repo by KhaosT refers to 11.3 for the API availability but it's the same ballpark, Apple is already shipping it as part of their Vision framework, under an obfuscated class name, and the code samples runs the model directly on macOS : https://github.com/KhaosT/nhcalc/blob/5f5260295ba584019cbad6...
And what if you re-do the hashes on a Mac with auto-backup to iCloud -- next think you know the entire offending database has been sync'd into your iCloud account :-/
The algorithm alerts a human, who actually looks and makes the call.
See also here: https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29...
All you need to do to cause trouble right now, would be to get a bad image, hash it yourself, make a collision and distribute that.
Let's say for the time being that the list hashes themselves will be server-side. You won't ever get that list, but you don't need it in order to cause a collision. You would need your own supply of CSAM to hash yourself, which while distasteful is clearly also not impossible.
That someone will be law enforcement, and they will get a warrant for all of your electronic devices in order to determine if you actually have CSAM or not. It's literally their job to investigate whether crimes were committed or not. Those investigations alone can ruin lives, even more so if arrests are made based on the tips or suspicions.
Does that mean that Apple employs people who manually review images known to be child pornography 9-to-5? Is it legal?
Yes, although I'm sure a sufficiently motivated attacker can obtain some CSAM that they are reasonably sure is present in the database, and generate the NeuralHash themselves.
> At that point, you are reduced to the threat present in every other existing CSAM detection system.
A difference could be that server-side CSAM detection will verify the entire image, and not just the image derivative, before notifying the authorities.
Remind us what the attack is here? The neural hash and the visual derivative both have to match for an image to trigger detection.
* The photo itself is benign.
* The photo’s NeuralHash matches known CSAM.
* The photo’s image derivative is not benign. It looks visually like CSAM.
* The photo’s image derivative matches known CSAM per a private perceptual hash.
The above, combined, could have a victim reported to NCMEC without being aware they were targeted. Since Apple operates on image derivatives, they could be fooled unlike other cloud providers. That is the claim.
At that point, the victim could point law enforcement to the original CloudKit asset (safety vouchers include a reference to the associate asset) and clear their name. However, involving law enforcement can always be traumatic.
Afaik the image derivative isn’t checked for looking like CSAM. It’s checked for looking like the specific CSAM from the database.
If someone wanted to plant CSAM and had control of an iCloud account, it seems far easier to send some emails with those images since iCloud Mail is actively scanned and nobody checks their iCloud Mail account, especially not the sent folder.
The question is whether the visual derivatives are checked against derivatives from the database or just against abstract criteria. That seems to be an unknown.
> However, it’s not clear to me that you can pull off all three simultaneously
Agreed. People here seem to keep assuming that you can, but so far nobody has demonstrated that it is possible.
"We're so excited to bring you all these new features and bugfixes in iOS 14.3, plus one more thing you'll hear about and object to in future. Too bad."
let scanFile4CSAM: Bool
if #available(iOS 16.0, *) {
scanFile4CSAM = true
scanFile4CSAM = is_iCloudPhotosFile && Device.Settings.is_iCloudPhotosEnabled
Edit: "These efforts will evolve and expand over time."[1]
That is, for each 96 bit neural hash value, there exist (on average) 2^3110304 unique input images that hash to that same value.
Again, these are of course trivial facts, which do not rule out that image recovery (in a "get back something that looks similar to the original input" sense) is possible, but you should be aware that "similar" to the network need not mean "similar" to a human.
> but you should be aware that "similar" to the network need not mean "similar" to a human.
With techniques like GAN and DLSS, it is quite possible to generate some photo realistic image being enough similar to the original one, or at least leaking some private information.
[1]: https://www.hackerfactor.com/blog/index.php?/archives/929-On...
EXCEPT... neural hash also claims to be robust to modifications to images that would result in a similar-to-human-image. If the 96 bits is enough to tag such similar-to-humans results, why couldn't a brute force approach yield such similar-to-humans images? Indeed, a nefarious person intent on producing CSAM could set-up something like a generational-adversarial system that the produced CSAM images using the hashes along with other clues.
96 bit is just not enough data to generate anything meaningful, just give up on that thought.
That someone is simply wrong.
- onnx.ai
I have never heard of this before and had to look it up. Is it widely used? Can I define a model in onnx and run it "everywhere", instead of learning pytorch or tensorflow?
ONNX is a representation format for ML models (mostly neural networks). onnxjs is a just a browser runtime for ONNX models. While it may be true that onnxjs is neglected, please note that the 'main' runtime, onnxruntime, is under heavy active development[1].
Moreover, Microsoft is not the sole steward of the ONNX ecosystem. They are one of many contributors, alongside companies like Facebook, Amazon, Nvidia, and many others [2].
I don't think ONNX is going away anytime soon. Not so sure about the TF ecosystem though.
There could be multiple reasons for the degraded performance:
- Are we comparing apples to apples here (heh), e.g. ResNet-50 vs ResNet-50?
- Was the ONNX model ported from TF? There are known issues with that path (https://onnxruntime.ai/docs/how-to/tune-performance.html#my-...)
- Have you tried tuning an execution provider for your specific target platform?(https://onnxruntime.ai/docs/reference/execution-providers/#s...)
> Resizing the image yields same hash, even down to below 200x100. Cropping or rotating the image yields different hashes.
It is not like a cryptographic hash, where altering a single bit will completely change the output.
What I had in mind when referring to 'basic evasion' was 'cropping or rotating', as per your original comment.
All that being said, I admit that generating adversarial examples for models with known weights is not a difficult task.
If two hashes are off by a bit or two, chances are that the two images are derived from the same, or similar, source image.
https://apple.stackexchange.com/questions/395508/can-i-mount...
So in the end we'll be left with a choice.
1. Allow Apple to scan your files.
2. Disable any kind of encryption letting anyone who steals your laptop access all your files.
Maybe I'm missing context, what files are you referring to?
Also I’ve edited /etc without disabling FileVault, is it just /System which is protected this way?
Even if they didn't have access to the original (for whatever reason), they train their own learning algorithm (supervised by their manual verification checkers) to detect the fake submissions.
Anyhow, adversarial attacks transfer reasonably well across models so you could create attacks on models you think Apple would use internally.
I’d imagine the first think Apple would do is put attack spammers on an ignore list. However, that would only work until the images start propagating on the wider internet via forums and social media.
The crowdsourced attack idea would also be contingent on thousands of people willing being flagged as pedophiles.
For crowdsourcing, I mean that the crowd is a distribution network and not the attack creator (e.g., they are a self-distributing virus/“worm”). The attacker takes those images (the initial “worm” programs) and uploads them to reddit as a catchy meme for worldwide distribution. Meme sharers wouldn’t be aware that the meme has a hash collision with CP.
The entire defense of this sort of thing is obscurity (further obscured by using “magical” machine learning), since nothing is proven about how collision resistant the algorithm is. At Apple’s scale, it’s as careless as rolling your own crypto hash.
I still don't quite see how the 'worm' would work in practice. It's not just a matter of sharing a link to an image, but you have to add it to your photo album. Maybe I'm getting old, but I don't have a single meme-type image in my album.
I don't think this can be classed as a security by obscurity problem. Security by obscurity fails because they 'key' (the obscure bit) can't be easily changed if it leaks. But given Apple has both the target and candidate images available to it, it can in effect generate a new key at the expense of having to do additional computation.
Same thing goes with the worm. Posting infected memes is one way and that’ll passively get some downloads. Another way is to text the pictures via bots to people and hope their SMS is iCloud backed up. The point is if someone figures out how to make an attack, they (or entity they sell it to) will almost surely spend great effort in social engineering the distribution.
I agree that it’s probably not the right term. My main concern though is that the class of attacks that is effective against one model will likely be effective against the entire class of similar models. So Apple’s main defense is the limited feedback loop caused by keeping part of the modeling private rather than a traditional computational complexity defense, which would allow disclosure of the entire implementation. It’s taken one week for the proof of concept to be demonstrated and most of that was likely boilerplate API code; it would take perhaps even less time to retarget if Apple disclosed the entire implementation with a different model. The demonstrated attack was textbook material, so it’s not unreasonable to believe that all attacks are textbook material.
For the folks who are interested in stopping abuse of children, there are many other approaches that would break the market for new abuse and new CSAM. This just isn’t going to move the needle and I have to assume they know that.
I’ve completely lost trust in Apple because I can’t understand what their motivations are. I _do_ understand the technology, so I’m pretty tired of articles suggesting this is some sort of misunderstanding and not Apple taking a giant leap towards enabling authoritarianism, and of course building literal thoughtcrime enforcement into end user devices, which is beyond even what 1984 imagined.
This was the minimal legal requirement or something.
The whole thing is baffling to me.
If they had led with that, we’d be having a conversation about the evildoers in Congress rather than Cupertino.
Seems like undue speculation. The government cares about far more than just CSAM. They care about terrorism, human and drug trafficking, organized crime, gangs, drug manufacturing, fraud etc.
This type of speculation only makes sense if Apple intends to expand their CSAM detection system to detect those other things, as well.
> This program is ambitious, and protecting children is an important responsibility. These efforts will evolve and expand over time.
I remember reading about the CSAM ring the FBI (IIRC) infiltrated and ran for a period of time, that group had strict rules on how to access and share material that, if followed, would have completely protected them but the majority of them were sloppy and got caught. Criminals really aren't that smart by and large. Will this catch the smartest of them? Probably not, but it will catch a good number I'm sure.
All that said, I'm not a fan of these changes, I just dislike arguments that don't hold water against it.
It feels like elaborate privacy theatre trojan horse to introduce in device surveillance.
[0]: https://user-images.githubusercontent.com/1328/129860810-f41...
[1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
If you think there is a credible mechanism, please link to it.
See: https://www.apple.com/child-safety/pdf/Security_Threat_Model...
Either way, if the claim is that it’s possible to reverse engineer CSAM from the hashes, proof is needed, and nobody has provided even a proof of concept.
The person I responded to was claiming it had been demonstrated. I asked for a link to evidence. You just made a hypothesis about how it might work. That’s not helpful.
This is basically rule 1 of testing neural networks: if the testing data is different from the training data and the results are still correct, your network is "reading" the data correctly and not just memorising a list of known values. I guess this means you'd also need to prove that the decoder doesn't turn most hashes of non-illegal images into illegal images, but if you also did that, you'd have a pretty strong case that the illegal data is in the hash.
NCMEC did, certainly, but I don't think Apple ever got the actual images themselves; just the resultant hashes.
Sure, it's possible, but that doesn't seem to have happened in the past decade of PhotoDNA scanning cloud photos to match hashes provided by NCMEC - why would it suddenly start happening now?
Server centric is the primitive that gives you periodic batch. Client resident let's you build up a real-time detection network.
Also, as they say in the financial world: past performance is not indicative of future results. No one would have thought to do so because this step hadn't been done. Now that this step has been done it is an easier to sell prospect. This is how the slippery slope works.
What's the realistic difference here between "my phone scans the photo on upload to iCloud Photos" and "iCloud Photos scans the photo when it's uploaded"?
Latency of upload doesn't come into play here because the scan results are part of the uploaded photo metadata; they're not submitted distinctly according to Apple's technical description.
(And given the threshold needed before you can decrypt any of the tagged photos with the client side system, the server side scanning would be much more "real-time" in this case, no?)
If it's happened, it's unlikely the public would know about it.
If you’re actually trying to stop abuse, having the computer create fake CP seems like an ideal outcome, since it would avoid the need for abuse of children.
Flooding the market with fakes and then directing consumers of the fakes to whatever mental health resources are available seems like it would fit the claimed problem far better than what apple is currently trying.
How much do you want to bet google will bring something similar to this to "keep up with the industry demands and partners requests". That would be the day either i go full lineageOS if they decide to not join the party or a dumb flip phone for ever. i will not subject myself to this because i know the government "WILL" hunt me down for being a dissident.
It's only on their servers that they will do the check against the database of CSAM content. So in that sense, it's pretty much the same that what other providers do, it remains attached to their online service, and they check the hash against the database instead of checking the picture (as others do).
If you don't use their iCloud service, the hash is never checked.
I still don't think having the client as part of the system is a good thing, but in terms of abuse it's about the same thing.
What Apple's system allow is a way to do a check while keeping the data encrypted in some 3rd party service. That part certainly raises questions should it be extended.
> "He was trying to get around getting caught, he was trying to keep it inside his email," said Detective David Nettles of the Houston Metro Internet Crimes Against Children Taskforce reports the station. "I can't see that information, I can't see that photo, but Google can."
"email" here is presumably Gmail, which Google owns and is responsible for. The dude was storing illegal data on Google servers in plain form. His personal devices were only checked by LEOs after a warrant was issued. Definitely not the same as Apple scanning the images on your device (not their servers).
https://www.apple.com/child-safety/pdf/Expanded_Protections_...
> This feature only impacts users who have chosen to use iCloud Photos to store their photos.
The only reason to scan clientside for a cloud service is to scan files that are not uploaded, or are end to end encrypted.
Apple already maintains an e2e backdoor (in the form of non e2e iCloud Backup) for the FBI and US intelligence agencies. It is extremely unlikely that they will e2e encrypt iCloud Photos.
I suppose had they not gone down this road, the headlines would have been "Apple makes it easier to share child porn online".
I doubt they'd bother doing e2e for iCloud Photos if they're intentionally not doing it for iCloud Backup.
Citation? I don't believe this is correct, or at least it's an incomplete assertion.
Assuming they do get with the iCloud backup, these keys would be inside the device's Keychain file which is encrypted at rest by the Secure Enclave. Thus even with access to a full, unencrypted backup of your iPhone, the keychain itself cannot be decrypted by Apple
(It can't be decrypted by you either, if it's restored to different hardware. This is why iCloud Keychain exists. And that is end-to-end encrypted.)
It's the difference between the airport checking my luggage for illegal drugs and the police showing up at every person's house once in a while to check if any drugs happen to be on the premises.
They're only scanning photos that you upload them to iCloud Photos - this is not (currently) a blanket "we'll scan all your local photos whenever" situation.
> It's the difference between the airport checking my luggage for illegal drugs
... and FedEx/UPS checking your outgoing packages for drugs.
Interesting USPS isn't mentioned there. Another example of 4th Amendment workarounds through private industry? Or are you just not aware of what the USPS equivalent program would be?
Genuinely curious.
You have to think big. You need people to open a package, inspect, and reseal it.
I can make a dropshipping business, turn a profit, and swamp your unboxers to the point you stop trying. I'm not out to do that, so I wouldn't bother, but there are ways, and you'll be surprised the hijinks you can get up to when adversarial attack of infrastructure or process is done right. Quite a lot of modern systems that just werk do so because on the whole, people don't spend much time being dicks. Unfortunately, my brain seems to have a part that enjoys the challenge in its idle time.
Measuring is hard. Especially when someone has their hand on the "noise" dial.
Actually, you're right. That's exactly what it's like. The former is how your Government routinely invades your privacy; the latter is an unequivocal violation of the Fourth Amendment of the US Constitution.
As it is with on-cloud versus on-device scanning. If Apple were compelled by the US Government to expand the on-device scanning to search for anything, that too would be an unequivocal Fourth Amendment violation.
Whereas any scanning that occurs on the cloud is not subject to Fourth Amendment protection. It's excepted under the so-called "third party doctrine", which effectively means the Government can rifle through whatever they want for any reason.