Hacker News Reader

Comment by marcinzm | Hacker News Reader

Given all the zero day exploits on iOS I wonder if it's now going to be viable to hack someone's phone and upload child porn to their account. Apple with happily flag the photos and then, likely, get those people arrested. Now they have to, in practice, prove they were hacked which might be impossible. Will either ruin their reputation or put them in jail for a long time. Given past witch hunts it could be decades before people get exonerated.

View on HN

>Given past witch hunts it could be decades before people get exonerated.

Given how pedophiles are treated in prison, that might be longer than your expected lifespan if you are sent to prison because of this. Of course I'm taking it to the dark place, but you kinda gotta, you know?

Someone is going to figure out how to make false positives, and then an entire genre of meme will be born from putting regular memes through a false positive machine, just for the lulz.

Someone else could find a way to make every single possible mutation of false positive Goatse/Lemonparty/TubGirl/etc. Then some poor Apple employee has to check those out.

If Apple is indeed using CNNs, then I don’t see why any of the black-box adversarial attacks used today in ML wouldn’t work. It seems way easier than attacking file hashes, since there are many images in the image space that are viable (e.g., sending a photo of random noise to troll with such an attack seems passable).

What's a CNN?

Convolutional Neural Network e.g., deep learning

If the process of identifying the images is done on the device, then a jailbroken device will likely give an attacker access to the entire DB. I'm not sure how useful it would be, but if the attacker did have access to actual known CSAM images it probably wouldn't be hard for them to produce false positives and test it against the DB on the jailbroken device, without notifying the company.

The "hack" might be very simple, since I'm sure it's possible to craft images that look like harmless memes but trigger the detection for CP.

The new and improved swatting.

Couldn't the hack just be as simple as sending someone an iMessage with the images attached? Or somehow identify/modify non-illegal images to match the perceptual hash -- since it's not a cryptographic hash.

Does iCloud automatically upload iMessage attachments?

Doesn't need to, the detection is client side at first.

No, like many others commenting on the issue, you seem to only have a vague idea of how it works. Only photos being uploaded to iCloud are being scanned for CSAM.

And you have an overly optimistic idea that they will not enable this feature more broadly. You really want to trust them, when this incident shows that they do not intend to be fully forthright with such changes?

They published full technical documents of what is happening and what is changing, and this is what this debate is about. It's a bit odd to argue that they are not forthright, this is all documented. They could have updated their terms of service vaguely and never mention that feature, they did not.

Then tell us. Because this is what apple says:

The Messages app will use on-device machine learning to warn about sensitive content, while keeping private communications unreadable by Apple.

Next, iOS and iPadOS will use new applications of cryptography to help limit the spread of CSAM online, while designing for user privacy.

https://www.apple.com/child-safety/

There is no ambiguity here. Of course they will scan images in the cloud as well, but they are explicit in saying that it is (also) on the device itself.

The operative separator is “Next”

Apple is announcing 3 new ‘features’.

First one scans iMessage messages / photos on device / warns kids and partners.

Second one is the CSAM photo hash compare in iCloud upload feature.

Third one is the Siri search protection/warning feature.

Stand corrected on the first part.

But surely iCloud upload feature is on the device. And if it was only in the cloud they wouldn't need to mention iOS or iPadOS at all.

But what’s the practical difference between scanning photos when they’re uploaded to iCloud on a server, or on device?

A world of difference. Both in practical terms and principle.

To start, once you upload something to the cloud you do - or at least are expected to - realize that it is under full control of another entity.

Because of that you might not use iCloud or you might not upload everything to iCloud.

I think you might still be confused? Only photos being uploaded to iCloud are scanned. So users can still choose not to use iCloud and avoid this.

I certainly hope you didn’t get yourself all worked up without actually understanding what you’re mad at :)

You are mistaken, the iMessage feature is for parental consent and is not used at all for the CSAM database.

It is not related to the CSAM database feature.

Read details here: https://daringfireball.net/2021/08/apple_child_safety_initia...

I think so, since the iMessages are synced across devices.

No, iMessages are stored on the device until saved to iCloud. However, iMessages may be backed up to iCloud, if enabled.

The difference is photos saved are catalogued, while message photos are kept in their threads.

Will Apple scan photos saved via iMessage backup?

I would assume yes, that this would cover iMessage backups since it is uploaded to their system.

This is already possible using other services (Google Drive, gmail, Instagram, etc.) that already scan for CP.

Does Google scan all files you upload to them with an algorithm like the one now proposed? Or do they have only a list of exact (not perceptual) SHA hashes of files to flag on? The latter I think is also used for pirated movies etc being removed under DMCA?

I'm pretty sure they use perceptual hashes for matching CSAM. A lot of cloud services do this sort of thing.

Yes: it’s called PhotoDNA and is used by many, many services. See https://en.wikipedia.org/wiki/PhotoDNA

SHA hashes aren’t suitable for this: you can change a single bit in the header to bypass a hash check. Perceptual hashes are designed to survive cropping, rotation, scaling, and embedding but all of those things mean that false-positives become a concern. The real risk would be if someone figured out how to many plausibly innocent collisions where you could send someone a picture which wasn’t obviously contraband or highly suspicious and attempt to convince them to save it.

This is really a difficult problem to solve I think. However, I think most people who are prosecuted for CP distribution are hoarding it by the terabyte. It’s hard to claim that you were unaware of that. A couple of gigabytes though? Plausible. And that’s what this CSAM scanner thing is going to find on phones.

A couple gigabytes is a lot of photos... and they'd all be showing up in your camera roll. Maybe possible but stretching the bounds of plausibility.

A couple gigabytes is enough to ruin someone’s day but not a lot to surreptitiously transfer, it’s literally seconds. Just backdate them and they may very well go unnoticed.

It's also possible to 'hide' photos from the reel in the photos app. Many people are unaware of that feature so an attacker could hide as many photos they want in your iCloud.

Assmuning they have access

With the number of security flaws that exist these days we should assume someone always has access that is not intended.

How do you do that?

Gigs of software updates and podcast episodes are regularly downloaded to phones without being noticed.

How frequently do most people look at their camera roll? I'd be surprised if it's more than a few times a week on average.

Does an attacker even need access to the phone? If iCloud is syncing your photos, your phone will eventually see all your pictures. Unless I've misunderstood how this works, the attacker only needs access to your iCloud account.

> I'd be surprised if it's more than a few times a week on average.

For me it's probably 5-7 times per day, but I also take a lot of photos.

I think a few times a week is probably low-balling it, even for an average.

I see my camera reel about once every few months. If I'm not taking a picture I don't see the reel.

The camera roll's defaults display images chronologically based on the image's timestamp. I've got thousands of photos on my phone going back years.

If you hack my phone and plant some photos with a sufficiently old timestamp I'd never notice them. I can't imagine my situation is all that uncommon either.

As others have said, people have a lot of photos. It wouldn't be too hard to hide them a bit from obvious view. As well, I rarely look at my gallery unless I need to. I just add a few photos occasionally. So maybe once every two weeks I look at my gallery, plenty of time to initiate that.

Why would they hoard it in the camera/iPhotos app? I assume that storage is mostly pictures taken with the device. Wouldn't this be the least likely place to find a hoard of known images?

Wouldn't this risk exist already, as long as it is uploaded to icloud?

You don't even need hacking for this to be abused by malevolent actors. A wife in a bad marriage could simply take nude pictures of their child to falsely accuse her husband.

This tech is just ripe for all kind of abuses.

That picture wouldn't already be in the CSAM database...

There's no point for this tech then. There's no way that they will no try to expand to detect the presence of porn and children being both present at the same time.

Reply on news.ycombinator.com