Comment by wpietri | Hacker News Reader

Comment by wpietri | Hacker News Reader

Parent Full thread

I think you're underestimating the degree to which companies have their own internal logic that has nothing to do with the external reality. That tendency increases the larger a company grows. If you're in a one-room house, you can always see outside. If you're inside a giant office building, mostly you see the building.

Did somebody raise the concern of push-back? I'm sure. But the moral questions around CSAM are something that was settled long ago internally. When I was at Twitter fighting abuse, the CSAM stuff was a separate group. My boss called them The Department of Mysteries because we almost never saw them or spoke to them. It was led by a serious person, an ex-FBI agent or something like that. They did what they did and we were all ok with it and grateful for it, because that shit is horrific and we didn't want it on our platform and we didn't want to have to deal with it ourselves.

My cousin was a PO for sex offenders, and one of our regular discussion topics at family reunions was how sex offenders were way more technologically savvy than a state parole department. How they really needed more help in making sure offenders weren't reoffending while on parole, while also not forcing them to just not use computers and phones altogether. If even I've heard this, I'm sure that Apple execs have heard it from law enforcement a zillion times.

It's also clear Apple put a lot of thought into addressing the privacy concerns for this. Technologically, it's sophisticated, impressive.

So I can easily believe the people at Apple said, "Sure, there are reasonable concerns, but we think we have addressed them." And that they're surprised by the level of sustained pushback.

It's worth looking into the privacy concerns that Apple was actually trying to fix here:

1. Photos (and other documents) are currently uploaded to iCloud un-encrypted

2. These photos are already scanned for CSAM after upload

3. Because the photos are not encrypted, at any point, any government can file a court order to release those photos.

4. The court order can require Apply to not notify the user, or the public in general.

5. [Speculation] Such orders might already exist and be somewhat common within Apple

Apple wanted to fix this and introduce end-to-end encryption on all photos uploaded to iCloud, but scanning for CSAM was non-negotiable (due to internal or external politics?). They must keep doing it.

So they implemented this big mess of a workaround to scan for CSAM before upload and attach a cert with a decryption key only to photos that match so that they could later human verify once a user had enough matches and weed out false positives (which Apple acknowledge will happen) before notifying law enforcement.

Because of the direction that Apple came from, and how much effort they put into designing this system to maximize privacy, they saw this solution as a large privacy win over the existing situation. It's not surprising Apple might have been blinded to the privacy concerns of doing AI scanning of photos on user devices, they were looking at it from the wrong angle.

Was there a recent announced that Apple will implement e2e encryption or is that speculation?

We know they were working on it several years ago, but abandoned efforts over two years ago due to FBI push objections[1]

Maybe they have restarted those efforts, but with some modifications (like this?). It's hard to tell since Apple is a very secretive company, but the exact design of this system does strongly suggest end-to-end encryption of photos. (Why attach a decryption key to positive matches if the photo isn't encrypted?)

I guess it's also possible this was a project kicked of years ago before they abandoned the idea of end-to-end backups and it lived on as a zombie.

BTW, while I see Apple's PoV here, I don't think I agree with it. I think I'd rather they stick with the status quo. I'm curious what other people prefer: un-encrypted and subpoenable iCloud photos, or on-device CSAM scanning?

[1] https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...

I have a preference for the on-device scanning + e2e Photos. The overall security seems higher, assuming Apple's claims about photos only being scanned during the upload process are true. (If I don't trust Apple to be telling the truth about that, it's not like I could trust them to not already be scanning all my photos, so...)

> Apple wanted to fix this and introduce end-to-end encryption on all photos uploaded to iCloud

Apple has not come out and said this.

This may be what people hope is the motivation, but I would think they would be immediately coming out to say this if it were the case, as it could be used as ammunition against some of the criticism they've been getting recently.

True.

But I'll point out their technical summary[1] explictly talks about attaching decryption keys to positive matches, which you don't need to do if there is no end-to-end encryption.

[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

> 1. Photos (and other documents) are currently uploaded to iCloud un-encrypted

No, they are encrypted, just not e2e. Apple holds the encryption key, though.

> 2. These photos are already scanned for CSAM after upload

No, they are not.

> 3. Because the photos are not encrypted, at any point, any government can file a court order to release those photos.

Not because they are unencrypted, but because Apple can decrypt them on demand, since they gold the key(s).

> 1. Photos (and other documents) are currently uploaded to iCloud un-encrypted

Incorrect. Photos and documents are encrypted both in transit and at rest. They are not E2EE, however - i.e. Apple has decryption keys.

> 2. These photos are already scanned for CSAM after upload

Apple does no scanning in iCloud. All of your data (except email) is stored encrypted and it cannot be scanned. Apple escrows the decryption keys, which are provided only to comply with legal requests for user data.

> 3. Because the photos are not encrypted, at any point, any government can file a court order to release those photos.

The second part is true, but the first is not.

> 4. The court order can require Apply to not notify the user, or the public in general.

This is simply the law. If there is no non-disclosure order, it's Apple's policy to notify.

iCloud encryption docs: https://support.apple.com/en-us/HT202303

Legal process docs: https://www.apple.com/legal/privacy/law-enforcement-guidelin...

Encryption during transit and rest really don't count as encryption, at least not in this context. Apple can decrypt them time to do things like scan them for CSAM. I assumed I wouldn't need to state the obvious.

And yes, Apple have been scanning uploaded photos since about 2019. At least, that's when they modified their terms and conditions to allow it.

https://www.macobserver.com/analysis/apple-scans-uploaded-co...

The author of that article was way off. First, his evidence is that an email attachment triggered a report, and we already know email is the only data stored unencrypted in iCloud. Email scanning is the most plausible explanation for the ToS update he is reading like chicken entrails.

If Apple were scanning iCloud Photos, one would expect there to be hundreds of thousands if not millions of reports to NCMEC (last year Facebook reported 20 million). Reporting is compulsory, the information is public, and Apple reported 265 last year. Do the math. Apple is not scanning Photos.

Having worked for a large corporate I frequently saw decisions that did not always make sense, with the justification being that the company was an innovator and market leader, and that they wanted to be like Apple and tell the customers what they wanted. Market research and such were not considered useful.

I am not sure whether to laugh or cry at that. "We want to tell customers what they want. And it turns out what they want is this half-assed thing that will advance the career of the executive pushing for it."

I have my issues with Apple. But they have such a deep tradition of user research and customer focus. When they tell customers what they want, it's often because they have really good data indicating it's what customers do want but don't totally know it yet.

Thank you for this perspective. I've never worked at an organization of this magnitude, so I am definitely lacking some perspective.

> It's also clear Apple put a lot of thought into addressing the privacy concerns for this. Technologically, it's sophisticated, impressive.

I'm not sure about this. How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc?

Apple says their protection against authoritarian governments is that an image must appear in two or more government CSAM databases to be scanned. So if a corrupt politician adds something to the database, it won't be scanned unless a different government adds the same image.

Now, will China follow this? Probably not. But Apple's defense there is that China could have directly ordered them to build this scanning tool at anytime in the last decade anyway. It's not like China has a magical new tool for invading privacy when they could have (and actually have) just ordered tools to be built as desired.

For sure. It's not clear to me whether Apple's approach is truly sufficient to preserve the privacy and safety of iPhone users that whole governments are out to get. But it definitely looks to me like they thought hard about it and have given it a go.

Maybe Apple already built the tool and was like "let's use this tool we already have but just targets specific child porn images". That actually makes them sound more reasonable.. kinda.

>Now, will China follow this? Probably not.

It's quite possible that countries would do the same thing interested parties on both sides are accused of doing with membership and voting in the International Whaling Commission: find countries that have no interest in the matter either way, and buy their support (or in this case, their addition of images to databases), which can be quite cheap when the matters don't actually affect the country.

Was cooperation between two contries considered?

And of course it's absolutely unprecedented for two governments to ever collaborate /s

"Five Eyes" ring a bell?

> I'm not sure about this. How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc?

As it's setup today it can't be. The list is the intersection of at least 2 jurisdictions, and is only checked when items are going to iCloud.

Of course this could change, but it's important to recognize what it is now vs. speculation about what it could become.

A perceptual hash by itself is neither sophisticated nor impressive. Apple could have just applied a client-side perceptual hash and auto-reported you and disabled your device. That is not what they have done, and pretending the solution's sophistication is limited to the sophistication of a percetual hash seems either incurious or disingenuous—I'm assuming the former. Either way, it makes it difficult to engage with your argument.

A searing and possibly legit indictment... what can I say...

> incurious

Hurts to read this. But maybe you're right: my mind kind of shut off when I read the abstract (client side scanning). I can't say I've looked into the details.

If you're not impressed by the perceptual hash, then you should at least be impressed by their implementation of the private set intersection. That's something that not a lot of companies could pull off.

It doesn't make the whole system any less fundamentally flawed, but I think the technical "Wow!" factor certainly helped many people be OK with it. In this view, the shiny new PSI system is a trojan horse for the totally unaccountable and opaque hash set that they've feeding into it.

> How is a perceptual hash sophisticated and impressive given that it can be abused by governments demanding Apple scan for political content, etc?

It's pretty sophisticated when you look at everything implemented and also consider the infrastructure / review pipelines that are required. See the link below:

https://www.apple.com/child-safety/pdf/Security_Threat_Model...

It's based on perceptual hashing, but the whole end-to-end system is clearly sophisticated when operating on Apple's scale.

This. I will add to this a little more. A big company is its own country. It has its own customs. It has its own ruling class. It has its own language.

All those things can make things interesting and sometimes misfire in a form of an idea that is not received that well in the real world.

Reply on news.ycombinator.com