There’s a crack in the iPhone foundation and it could get a lot worse
macworld.com
macworld.com
> While today it has been purpose-built for CSAM, and it can be deactivated simply by shutting off iCloud Photo Library syncing, it still feels like a line has been crossed
Two simple facts:
(1) The system, as described today, isn't any more invasive then the existing CSAM scanning technologies that exist on all major cloud storage systems (arguably it's less invasive - no external systems look at your photos unless your phone flags enough of your photos as CSAM, which brings it to a manual review stage)
(2) Auto-updating root-level proprietary software can be updated to any level of invasion of privacy at any time for any reason the provider wishes. We aren't any closer to full-invasion-of-privacy with iPhone than we were before; it is and always has been one single update away. In fact, we don't know if it's already there on iPhone or any other proprietary system such as Windows, Chromebook, etc. Who knows what backdoors exist on these systems?
If you truly believe that you need full control and a system you fully trust, don't get a device that runs proprietary software. If you're okay with a device that isn't fully trustworthy, but appears to be benevolent, then iPhone isn't any worse than it was a month ago.
Until there's evidence otherwise, iPhone will continue to be as trustworthy as any other proprietary closed-source system. If you need more than that, please contribute to projects that aim to produce a modern, functional FOSS smartphone.
The mental and ethical lines are clear when I hand content to a server, that's a contract I can understand and agree to. When it runs on my device, even if it's because of the option to send it to the server, that feels fuzzier. As Ben Thompson put it it's the difference between Capability and Policy.
This is now rubbing in our faces that the privacy is only in the policy, not the capabilities of the software. And the capabilities of propriety software are pretty knowable, through a combination of inspection and long term observation.
The end result isn't any different today, but it highlights the political position that Apple and others are being put in. I honestly don't fault Apple for this on anything except their clearly poor messaging, but they are getting pulled from all sides by governments who don't like the idea of them actually implementing a fully closed and end-to-end encrypted system for user data.
Strongly agree. A law requiring police cameras in rented trucks is an invasion of privacy and a loss of rights for the populace, but it's a whole lot better than having those cameras in your home.
But the sad truth is that handheld devices are actually more like rented tools than they are private homes. Apple devices are windows into SaaSpace; their local storage and compute capabilities are merely implementation details. They aren't anything like the PCs of yore.
I agree, but it's frustrating to me because it's an illusion: if your device is running proprietary software, you don't control the device. At that point there's little difference between your device and a server in the cloud, aside from technical characteristics like latency and compute power.
Some of this was just due to being further behind on the SaaS ramp than MS and Google. Everyone expects a Google-services enabled Android device to basically be a cloud thin client, but less so an iPhone. And Apple has actually stood up for encryption and not putting in openly exploitable backdoor capabilities just to appease law enforcement, see their back and forth with the FBI a couple of years ago.
Apple seems to be coming up with awkward solutions as they try to do the "right thing" in some places (more end to end encryption) while governments / law enforcement don't want hidden data because of "the children", terrorism, etc. They announced this proudly as though they had finally found the holy grail to negotiate this quagmire but the messaging did not go over at all how they seemed to expect.
I don't think there's a path forward for fully open and non-proprietary software that everyone uses. It hasn't happened on the desktop so I doubt it will on mobile devices. And most people want their devices to be cloud terminals, not sovereign system states.
So I think it comes back down to a legal and political issue of what requirements governments put on these tech companies, how data and privacy protections are guaranteed (or forced to be broken), and how that gets enforced.
This system will only work on photos you are going to upload to icloud photo library. If you dont enable this, no picture will be scanned. In my view this is exactly the same as before where this scan was running on the server.
I have no issue with people who give of very very basic user freedom because they want to opt into the iOS "ecosystem", but I don't see how this particular thing would push anyone of the ledge.
Once this system is in place, it takes only 1 tiny adjustment/exploit to scan other private stuff on your devices, which you would never upload to a cloud.
This has always been true, there's no "Once this system is in place" qualifier necessary. This is the reality of running someone else's proprietary code at the root level on your device.
Correction, scanning cloud destined files, as a step of the upload pipeline. Local files not destined for the cloud are not scanned. It's a scan happening before the network stack of the pipeline, rather than after. But it still is already in the pipeline, an its way, at the point of scan (at least that's my understanding of their docs).
All HN users in this thread < All tech journalists who are writing about this topic < All iPhone users
Right, my point is that the latter (scanning of offline data) does not yet have any evidence of occurring, and we aren't "closer" to that being a reality than we were before: it is and always has been one software update push away from being a reality.
> we aren't "closer" to that being a reality than we were before
I want to believe this but it feels very much like we are. And I don't think it's purely psychological or alarmist.
> it is and always has been one software update push away from being a reality
As a software engineer it's hard to shake the feeling that we're now just a patch release away from misuse, where as last year we were a major update away.
It's still only “one software update push away” either way if you're counting releases, but it's not if you're counting tickets in a product backlog. It's easier for governments to expand an existing feature than to pressure companies to build one from scratch. Apple already built the feature security services dreamed of even though it had no legal obligation to (it has to report CSAM it finds; it does not have to actively search for it on your device).
It's also easier to normalise the use of client-side scanning when one company has shipped it (without talking to the rest of the tech industry, and after declining invitations to talk to Cyber Policy/Internet Observatory teams at Stanford who are trying to help the industry as a whole with an improved collective approach). That pressure and the likely additional implementations of client-side scanning we'll see further expand the potential for abuse.
I really enjoyed the Stanford discussion available on YouTube[1]:
> @8:07 > …the other issue being that it wouldn't be terribly difficult to expand this client-side system to do all of the user's photos so we kind of have to trust the software to only apply this to things that are backed up to iCloud.
> @17:24 > I think it's inevitable that some government — quite possibly the U.S. — is going to want to expand this to counter terrorism and, well, I think for a lot of people that sounds reasonable. The problem is that the definitions of terrorism are somewhat malleable…
None of this feels alarmist to me. Yes, proprietary software is only ever an update away from throwing us all down the slope. But the slope is not a pure binary thing that is or isn't — companies can increase the gradient by decreasing the work it would take to hurt us, and that's what Apple has done here.
I don't agree. Putting aside the "NeuralHash" algorithm that Apple made to try and match CSAM, scanning the files on a device is remarkably simple, and something Apple already does as part of your device's standard functioning (indexing files, attaching labels to pictures based on content detected by AI models, etc).
Apple could have already implemented a secret image tag for "terrorist material" or "drug material" that is attached automatically to images, hidden from the user, and phoned home / reported to FBI when a threshold is met. How would you know this system doesn't already exist? Literally all the components for this system were already in place.
> Today marks the official public unveiling of Expanded Protections for Children, and I wanted to take a moment to thank each and every one of you for all of your hard work over the last few years.
So this doesn't sound like a feature where all the puzzle pieces were in place and they just needed NeuralHash.
Even if you casually “put aside” NeuralHash as you suggest, the amount of research and testing that has happened to ship the system they've described is not trivial.
I stand by the idea that this was never a point release away.
> Apple could have already implemented a secret image tag for terrorist material" or "drug material" that is attached automatically to images
Apple cannot even consistently tag cats and dogs. There is no way it was ready to ship a feature that tags drug or terrorist material in a way that generates few enough false positives that agencies won't just turn it off.
I do agree with you that we have no way to know what's running on closed-source devices (or even open-source ones, unless we personally audit the whole process from dust to device).
For me, though, “you can't ever really know what's running on your device so why care about contentious new things you've just learned will definitely be running on your device” is not compelling.
I might swallow 10 spiders a year in my sleep, but if someone offers to feed me one I think it's fair to decline.
Except for those cloud storage systems that have offered client-side encryption from day 1.
Pretending that server-side scanning is something naturally given and should be accepted as baseline biases the arguments that follow.
Their 1-in-a-trillion estimate of false positives is very likely total horseshit (there was a recent article on dissecting that claim from someone familiar with CSAM).
It also opens up a "swatting" avenue where if someone hacks your phone they can upload CSAM images and apple will sicthe authorities on you. Good luck with that.
Really we need to bring back comp.risks and start training ourselves to think a bit more cynically about the downsides of technology.
Sorry, you can't install this bit torrent client or this crypto wallet...
If you don't want your photos scanned, turn off iCloud. You can disagree about whether or not that's anti-consumer, but you cannot argue they are changing anything about your device or it's EULA (they are changing the EULA of a service).
Next will come scanning of your phone's content. "Hey, if you never send email or chat, the scanner never runs."
And keyboard logging. "For the children".
Encrypt end-to-end all you want, it doesn't matter if your device is running spyware.
We are in agreement.
It's a feature of Communication safety in Messages. This is opt-in.
I don't know the details here, but you can disable iMessage and just send plain SMS, which in theory does not send the data to Apple. But then, part of the point of using iMessage is that the message contents are not sent to your carrier and have some encryption in place. I would guess your best bet is to find another phone manufacturer and use a different messaging service that provides better security if you want to opt out.
> It’s also worth pointing out that it’s a feature of the Messages app, not the iMessage service. For one thing, this means it applies to images sent or received via SMS, not just iMessage. But more importantly, it changes nothing about the end-to-end encryption inherent to the iMessage protocol. The image processing to detect sexually explicit images happens before (for sending) or after (for receiving) the endpoints. It seems like a good feature with few downsides. (The EFF disagrees.)
https://daringfireball.net/2021/08/apple_child_safety_initia...
The child protection part can be enabled for under 13s if they’re in a family account. If enabled messages app will try and detect adult images being sent and received and give a warning to the child, it can also let the parents know about it.
I had quite a hard time explaining to some people why the anger towards this wasn’t pro-pedo essentially because they couldn’t understand why anyone would be against these people being found.
You don't want Google? No problem, blow the ROM away and install Lineage without GApps, you can even mess with MicroG if you're reliant on some app that needs Play Services.
Corporations and governments combine to basically make privacy a niche thing for tinkerers only, you simply cannot mass-market the development attitudes nor technical and opsec skills needed to achieve a real degree of privacy on the user side.
The solution to this problem does not lie in a technical plane, nor does it in digital escapism.
Grandma does just fine on Android and would be utterly confused by iOS.
It's all about what you're used to.
I want an option that is "grandma" compatible. In other words, it isn't really viable if you can't just buy it off the shelf, or have to dig into settings to opt out. Right now if I go to Best Buy (or insert your favorite retailer here), I can't buy hardware off the shelf that won't send my data to a remote server by default.
A UI, sure, I could understand that, but a kernel?
I'm not too keen on contributing to linux monoculturization of IT anyway.
Shitty UI design is irrelevant of OS.
The FBI/CIA/CCP are not the people we elect.
My representative is behind the charge to undo the app store monopoly. We can also push for less surveillance, less centralization of power, etc.
It seems if the OS vendor doesn't get to have a default browser, default app store, and default cloud store, that they don't get access to scan your files without asking either.
Legislate.
The only option is to abandon ship.
Subaru lets you do it over your Wifi, but does not seem like they come with a modem:
https://techinfo.subaru.com/stis/doc/ownerManual/Gen4_FOTA_H...
Although, Mazda’s website does say this, so maybe it will not be for long before they all come connected to mobile networks:
https://newsroom.mazda.com/en/publicity/release/2021/202106/...
> Mazda intends to fortify our initiatives of development of fundamental software technology in order to be able to accommodate for next-generation Mobility as a Service (Maas) and update vehicle functions Over the Air (OTA)
> Five Japanese OEM companies 3 including Mazda will jointly develop standard engineering specifications of next-generation in-vehicle communication devices to push for a standardized communication system in order to provide safer and stress-free connected services sooner.
The next step in that debate is "yeah but the big monopolies are making it impossible for a little guy to get in." Which is true.
We can agree the regulatory capture is bad.
In the meanwhile, Google is not openly saying they will run ML on your images on your phone. With Android, you don't have to sync to the cloud, and you could even replace or add your own camera option. You can side-load without jailbreaking, etc.
Now, not that most consumer friendly option, but the advice for this crowd is still good - if you still have an iPhone and this is the last straw for you, there are plenty of good options that still exist, today. And then - let's fight regulatory capture and big government so a more dynamic marketplace can take root.
The assumption that a "functioning" market will do a good job of catering to even fairly popular wishes does not seem to hold true in the real world, including for cases in which I'm pretty damn sure it's not some kind of government interference causing it not to. It's utterly common for plain ol' commodities subject to no special government scrutiny or control and with many suppliers to provide no option for features or product-types that would surely have many buyers, simply because no-one expects the returns to be as high as doing something else with the same capacity.
AFAIK this happens for a bunch of reasons, including that information is very, very far from being perfectly shared in all parts of the market, that there are significant costs associated with quality information-gathering, and efficient use of capital tending to cause production to cluster around tiny little bits of the possible product space (similar to how pharmacies like to build right next to each other, rather than spreading out to reduce travel-time-to-a-pharmacy in an area)
The point is that we're REALLY playing with fire here.
It's not. That's the problem.
Nothing anywhere says that Apple scans the pictures you store on your own device. They only scan the pictures you upload to their cloud service.
The scanning is done on device before upload in Apple’s case, and in the cloud after upload in Google’s case, but either way it is only done to photos that are uploaded to their cloud services.
If you are really claiming Apple scans the photos you don’t upload to iCloud Photo Library, then you are lying or dissembling.
I assume that is not what actually what you mean.
Now you appear to be making a false statement about your own words.
You only mentioned photos you store in your own device, and you said nothing about uploading to servers.
Here’s what you said:
> Nothing in those links says that Google can scan the pictures you store in your device using your own device to do the scanning.
BUT THE CAPABILITY TO SCAN CONTENT OF PHOTOS ON DEVICE EXISTS. The argument is tomorrow they can simply start sending scan meta data or captions of image content up to a server without you opting into cloud storage.
The capability exists on device. It's all baby steps.
As to the capability on the device, the capacity to scan for CSAM is very narrow and is very hard to repurpose.
The capacity to upload images to iCloud Photo Library on the other hand has been there for years.
At any time Apple could add some other kind of scanner if they want to, and there is no reason for it to use this mechanism. It would be terrible if they did but it has nothing to do with this.
Anyone with programming experience can tell you that if all they wanted to do was check arbitrary files against a list of hashes, they would be a simple mechanism to write.
There is no way that this mechanism helps them scan for other things. It isn’t even a step in that direction, let alone a baby step.
I'm not an expert in how CSAM works, but if it only as a block list against certain images it will be very ineffective.
The way I would expect it to work, is to recognize the content of images. The SOTA on this is pretty impressive. Knowing the content of the images is what Google does in the cloud, and its great. I can search my images for "Green taxi" and it will find it. Water. Sunsets. Anything.
If apple is introducing the ability to recognize photo content on the device (even if right now it is destined for the cloud as a pre-scan), it doesn't really matter that the model is only tuned to find child abuse, as an example.
Tomorrow, the hyper-parameters or the ontology could be expanded to search for anything. Political affiliations, location and timestamps (this doesn't even need modeling!), illegal objects or substances, etc.
The deal is that Apple is saying "we are going to use your device to determine what is in your pictures." The circumstances and scope of those determinations can change, but the expectation that Apple will be doing it is now publicly established.
You don’t have to be. It’s easy to find docs on this exact technology with Google.
> but if it only as a block list against certain images it will be very ineffective.
It is not.
> The way I would expect it to work,
You are basically completely wrong. Read up on it and then we can discuss it.
Even assuming this other shoe drops, then this logic implies that there is a moral duty to implement other types of scans to prevent other serious crimes. Surely Apple shouldn't abdicate its moral duty to prevent murder. There's really not much of a limit to this argument. As long as there is the level of false positives is similar to that of the technology now being implemented, there is a moral duty for Apple to deploy the surveillance methods.
Over 70,000 Americans die from overdoses each year, which is nearly double that of yearly vehicle fatalities. If Apple is already scanning photos and messages, why not save tens of thousands of lives while they're at it by detecting heroin and fentanyl dealers, too?
The camera is facing the wrong direction...
> But would it be able to say no to China?
In order to keep selling on the Chinese market, they already agreed to host their Chinese customer data inside China with a local cloud provider, and censor content according to Chinese requirements, and fully comply with the CCP requests. So once the "tank man" photo is added to the list of banned pictures, there's no doubt they'll happily comply.
In other words, "just the tip".
This move by Apple is the proverbial tip. Eventually you can bet Apple will let governments upload not just hashes of CSAM, but "hashes" of faces they'd like to monitor. Or objects in images (maybe weapons, or chemicals, etc.)
I am so glad I don't use any Apple products.
We need more options than just Android or iOS. Sure Android is "open sourced" but we all know its limited without all the Google things also installed.
We need a true third competitor.
Having had time to cool off, my response to this is to continue using Apple devices and supporting their privacy-focused way of CSAM scanning rather than the worse alternatives.
BUT, I'm going to start buying and supporting alternatives in parallel, and eventually start contributing to them.
If my biggest concern is government overreach, yet I think that we're "not there yet", then its incumbent upon me to be a small part in helping to make sure the alternatives are ready once we do "get there".
My reasoning being that in the FOSS world there's not a single company you can pump funds into (Pine64 does not proft much or at all from their devices, and Purism is just one company, not yet shipping the phones in large quantity), but individual developers scattered all over the world DO contribute and DO need our support.
It's the only way, until we get more Pines and more Purisms (minus the drama).
Is this correct? I have just read conflicting things about this online, I think even Ben Thompson wrote that it runs on the device too?
Bit of a shame... I wish it just sat elsewhere on their server before cloud syncing and then I could just disable cloud sync + use the phone without worrying.
What is the basis for this assumption?
The system only works with iCloud Photo Library, it needs a server side component to continue the process.
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
This is a very interesting read and personally I think they’ve gone to extreme length to make this system as private as it could be.
" The authorities are now moving not just to reduce the influence of foreign tech companies but also to force them to promote Russian services — as with the new regulation mandating government-approved apps on all new smartphones.
Apple has agreed to this "
Care to elaborate? I am genuinely curious
> Government critics have been targeted; Navalny’s Live Journal blog, which published investigations about corruption in Russian politics, and other political opposition sites were blocked. (Roskomnadzor said they were banned for calling on people to illegally participate in mass events).
>The “sovereign Internet” law required Internet Service Providers (ISPs) to install Deep Packet Inspection (DPI) equipment, which has been used by some countries, like China, for censorship. DPI equipment enables Russia to circumvent providers, automatically block content the government has banned and reroute internet traffic.
>It has required search engines, including Google, to delete some results...
>some have buckled under Roskomnadzor’s threats to block them if they don’t comply with censorship orders. In 2018, Facebook-owned Instagram, which has 54 million users in Russia, complied with the regulator’s requests to remove posts connected to corruption allegations by Navalny. In a tweet Navalny accused Instagram of submitting to “illegal censorship orders”. “Shame on you Instagram!” he wrote.
>After Roskomnadzor threatened to prosecute social media sites for encouraging minors to join the January protests, the regulator said TikTok deleted 38% of its related content, while YouTube and Russian social media site VKontakte removed half.
All from the link I've provided above already in the previous post: https://time.com/5951834/russia-control-internet/
The statement about DPI is wrong though: DPI is ineffective without government-mandated root CAs.
To me, then, the whole thing at least partly rests on those questions. If CSAM is a moral panic, then this technology is not just bad, but doubly so. Apple got played by the Feds, or maybe they really have been dreaming of backdooring their devices and finally had the rationale they needed.
But if CSAM isn't a moral panic but an actual growing problem that needs a technical solution (or assist), what are we to do? Just shrug and say "well it's a hard problem, maybe one day we'll solve it" and hope for the best?
https://tutanota.com/blog/posts/wer-wird-belauscht/
"Politicians regularly claim that they need to ban encryption to protect the children. But who is actually being monitored?"
Apparently by orders of magnitude over any child related issue, it's drugs.
That seems to imply that it is a "moral panic". If they said they were doing for drugs, or drugs + other stuff then that might be more convincing.
Moreover if the article is to be believed - such mechanisms are used at hugely higher rates for things they are not pitched as being for. So that breaks the narrative of 'we'll only use it for X'.
I'd love to see this problem analyzed in terms of layers. My sense is that one barrier is walled-garden apps. I'm happy to give these up. But I also have the sense that there are hard barriers involving firmware (especially cellular modems?), identity (SIM cards are designed to contain secrets you can't access), and connectivity (you can't decouple a phone number from its client as easily as you can with an IP address).
Today it's scanning your photos. Tomorrow it will periodically turn on the camera, do on device processing to check for child abuse, then only send the recording to FBI if it thinks so. Otherwise the recording will be deleted.
Once you cede control over your property, this is bound to happen sooner or later.
> But tools are neither good nor evil. Apple has built this tool for a good purpose, but every time a new tool is built, all of us need to imagine how it might be misused. Apple seems to have very carefully designed this feature to make it more difficult to subvert, but that’s not always enough.
> Imagine a case where a law enforcement agency in a foreign country comes to Apple and says that it has compiled a database of illegal images and wants it added to Apple’s scanner. Apple has said, bluntly, that it will refuse all such requests. That’s encouraging, and I have little doubt that Apple would abandon most countries if they tried to pull that maneuver.
> But would it be able to say no to China? Would it be able to say no to the U.S. government if the images in question would implicate members of terrorist organizations? And in a decade or two, will policies like this be so commonplace that when the moment comes that a government asks Apple or its equivalents to began scanning for illegal or subversive material, will anyone notice? The first implementation of this technology is to stop CSAM, and nobody will argue against trying to stop the exploitation of children. But will there be a second implementation? A third?
I’ll add this: Is our outrage at Apple enough to stop governments from requiring that they implement features like this?
Don't you see that it's two sides of the same coin, and end-to-end encryption can easily enable any sort of privacy coins and anonymity? We as advocates need a consistent position, otherwise we just have a gaping double standard.
If you're worried about anonymous transactions in crypto, because they can lead to, say, money laundering or financing terrorism, why are you not worried about end-to-end encrypted communication by those same terrorists?
But AAPL’s actions of late have me wishing there was a 3rd option for a phone.
They could have implemented true E2EE but they chose not to in deference to law enforcement and they could have fought the battle there. The amount of speculation being done on Apple's behalf here is baffling.
I think Jason's take is pretty measured and it's pretty clear that he's uncomfortable with this, especially given his long history with reporting on Apple but he is also pushing this false dichotomy.
Now, I really doubt the only reason they didn't implement E2EE was because of government pressure because making E2EE an invisible feature (which Apple loves to do) is a difficult task. I will however plainly state that the option of creating spyware to scan images on your phone with E2EE on the cloud or neither of these features being able to exist is a false dichotomy that is being pushed as a narrative by a lot of people.
>Apple’s Head of Privacy implies that it’s because it was ready, but that’s a bit of a dodge—Apple has to choose what technologies to prioritize, and it prioritized this one.
Apple didn't have to prioritize this ahead of anything in order for it to be ready to ship. They could have been working on it in the background for some time.
Conversely, how might Apple implement non-repudiation of user actions? If they control the platform utterly, how can they prove that a user really did something?
Why can't it run in cloud only when a device tries to upload something.
Then users can just turn off using the cloud services and retain their naive notion that what's on their phone is really private.
When are we doing to see a cloud service provider that matches iCould and other service providers but promises total anonimty
#NotUpdating #NotUpgrading #BoycottApple
Why did Apple do it, though? To create precisely this kind of cognitive dissonance. It's not Apple that's observing you. It's the government. Apple has basically created an API for government surveillance and can now say "see, we don't collect the data. We don't even know what big brother wants to know about you."
Apple just side-stepped the question on how to reconcile trust in propietary software and surveillance requests from the government. And everyone who tells you that it's not guaranteed that this feature will be extended to texts, spoken language, and other contents, is not worth talking to.
We now live in a world, where our personal computers come with "AI" that detects illegal use of those computers and automatically turns us in to the authorities. This general pattern won't disappear, ever. In a couple of years you will have to carefully consider every word you say in a car, every movie you watch, every website you visit, every program you write, because some neural net will use it to create a fingerprint that might be classified as illegal behavior.
BTW: I've been an Apple fanboy since the mid-80s, but I find the whole concept of my phone suspecting me of child pornography appalling. It's the thin end of the wedge, and IMHO Apple have made a huge mistake.
This is the biggest worry. Apple essentially waved their hand in their FAQ and said something to the effect of they’d never do something like that.
Yes you would. You’ll get called to congress and told in no uncertain terms what you’re expected to do, you’ll say no, the law will be changed, then the gentleman in a suit that people don’t say no to will call to your nice headquarters for you to walk him through his shiny new surveillance tools.
Once the tool exists, they’ll add whatever features governments in major economies tell them to or they’ll be forced into adding them.
All this said, I don’t blame Apple alone here and I have mixed feelings about it still - as I really do believe in their fight against CSAM. I just don’t think this is the magic solution to solving CSAM while preserving an innocent user’s right to privacy that they’re selling it as.
I’m also certain there are engineers who worked on this system who also share these concerns.
I also feel it's important to note that every major image host is doing the same thing Apple is doing, but server-side on unencrypted images. What Apple is planning to implement is, out of context, an improvement; in the same way that Google's FLoC is an improvement over ad tracking based on third-party cookies. Get a trusted client to cooperate in some smaller act of self-surveillance and we don't need the server to do full take.
That being said, I don't think we even need to rely on governments threatening Apple into doing surveillance. The scanning scheme relies on paying human moderators to basically look at CSAM all day and check if it's a valid match. Big tech moderation is already understaffed and overworked as is - so it's almost certain that the people involved will just forward everything as their sanity degrades. Apple will almost certainly wind up accidentally or unknowingly forwarding along bogus CSAM reports, no coercion required.
In The Cuckoo's Egg, Cliff Stoll tells about having his computer page him when it detected a hacking attempt. Telling about one time when his pager went off, he said, "My computer wanted me."
Now we have so many notifications and suggested actions from our phones and computers, it feels like we have become computer peripherals. Our computers want us to do things. They have agency, and we're responding to it.
But at least we still had agency, too. The computer had the initiative, and we could choose to respond, or not. Now, though, the computers are limiting our agency. They are becoming the controllers, and we the controlled.
I don't think it's just that Apple has made a huge mistake. I think humanity is making a huge mistake. We are allowing tech to turn us into children.
It's been a decade and a half of software not being the user's agent and instead some company's agent that in some cases you buy the hardware for.
Unless how we talk about software changes this is just the world we live in now. How many apps do you have that are genuinely extensions of your intent as a user and not company kiosks? Of probably about 100 apps on my phone I think I have maybe 5.
My biggest fear is the only reason I've seen this come up this week is the Apple community, that built me into a programmer from a waiter, has become susceptible to the same kind of irrational tribalism we see everywhere else - "surely this bad thing is happening because it's part of a secret plan to fix the other bad thing!"
Another article recently points out that even the md5sum databases include bad matches (a rhesus monkey was mentioned).
Not everyone is able to do this, but you're technical enough to be spending time on Hacker News.
(Also, this doesn't preclude lobbying for restrictions on government and corporate surveillance. It's not a dichotomy)
On my desk right now there are five Android phones (two of them don't even have SIM cards). I use them for different things. I wouldn't dream of signing in to all of them with the same Google account.
If you want to retain some degree of privacy and use Instagram on your phone, you need to get another phone for that.
They're cheap.
Second, phones are a terrible platform for privacy preserving technologies, both currently (mobile OS dumpster fire) and intrinsically (lower resources, poor input, well-known network identity). Anybody who cares about digital privacy should have a real computer laptop/desktop that they use for the bulk of their activities.
Not being locked-in into apps is critical nowadays.
Someone earlier astutely stated the metaphor “the thin edge of the wedge”; the movement to the “cloud” via the open web over the last decade was the thinnest part of the wedge.
Even to the extent web is open today, I’m able to run Linux on all my PCs/laptops and not to have an extra one to run proprietary OS in order to participate in modern life.
If we don't want to live in a surveillance-based dystopia, we need to push back against it politically as a society, not play cat-and-mouse as individuals.
And for that to work, we have to counter the usual "but think of the children!!!" narrative with one of our own - prevent CSAM creation at the source. I have written about how to get started previously: https://news.ycombinator.com/item?id=28114076
I think one can arrive at the opinion they would prefer not to have their recommendations flooded with stuff from from alt-right or conspiracy Youtube/Twitter/whatever without buying into propaganda.
Meanwhile, groups/people/movements that completely match that description are real and it's handy to have a term for that behavior, even if you think it's applied too broadly (and maybe it is). We've got the similarly-abused word "cult", but abuse aside it's still handy to have a way to refer to groups that actually do closely fit the colloquial, pejorative meaning of the term, and there's no way you're going to get people to stop using the word that way.
It should disappear due to active push back because freedom, democracy and human rights cannot coexist with such surveillance.
Privacy is where independent thinking is happening and generated and if it's not guarded properly the whole idea of democracy is not possible. More on that here : https://news.ycombinator.com/item?id=28084578
But to convince Apple there needs to be plenty more articles about CSAM and where it leads.
Everyone here is hating on Apple, but looking at what Peter Thiel is involved in - oh, that's FINE. He is rich, he is better than you, obviously. Don't believe me? Watch the Peter Thiel shrine that is HN downvote this.
https://www.vice.com/en/article/5db4ad/google-bans-safegraph...
Apple has just created a product that slowed the government down and they wouldn't stand for it and so they compromised.
But America talks a good talk about pushing back against surveillance but if it doesn't affect people day to day, no one will care.
Hell, almost a million Americans are dead and about half care more about being able to go to a bar than the easily preventable deaths.
Corporations have done nothing but repeatedly invade my privacy, exploit my data and attempt to squeeze money out of me at every turn. The relationship has seemed adversarial for quite a few years. So what should i expect from Apple here?
It sort of feels like the alternative is for Apple to stand up to governments. To fight for the people.. But this seems super, super hopeful. A desperate plea founded in wishful thinking that corporations have any incentive aligned with ours.
So what in your eyes is the right solution here? Apple letting us down here is par for the course here to me. But i also don't see an alternative where i'm not just blindly hoping some corporation will defend my freedoms for me. I've lost all trust in literally every other corporation.. honestly, should i have trusted Apple?
Today it's child pornography. What about tomorrow? Let me make a case. First, let me make something clear at the outset. I am 100 percent against child pornography. For all I care, castrate all the adults involved, when you catch them.
Now, with that made clear, let's take a look at how bad, comparatively, pornographic photos of underage individuals on a person's phone is—and my apologies to those who cannot stomach the following discussion.
The photos have already been taken. The harm here is indirect. The harm is that possession of photos means that you have become part of the market for child pornography, and provide incentive for the production of future materials. The direct harm occurs when the materials are being produced, and the direct harm is categorically worse than dissemination and possession of the photos.
Compare that to something like terrorism. Recruiting terrorists and planning terrorist activities is, I would argue, a greater and more direct harm than mere possession of child pornography. (Leave aside the production.) So, how can someone accept that this is okay for child pornography and not terrorism? And once we've accepted it's okay for terrorism, who is the "potential" terrorist? What is terrorist contraband?
And now that we've accepted sniffing for terrorists, what's next? Election "manipulation" that "threatens" our democracy? Public health "misinformation"?
This is the Panopticon.
Our latest Antipope.org post (discussed here yesterday), suggested we extend the child-pornography treatment to possession of unauthorized cryptocurrency.
https://www2.palomar.edu/anthro/animal/animal_3.htm
How could anybody be against criminalizing it!
Interesting. I’m not familiar with a criminal justice system with a false positive rate lower than NCMEC’s perceptual hashes.
I doubt it would suffice. Chemical castration, perhaps.
Is this a viable option when governments are pushing them to do something? To give governments access, etc?
Ie why should Apple fight governments here. They seem to have next to no incentive to do that, in the capitalist sense. Our society doesn't typically financially support moral stances, so is fighting the government, which "doing nothing" surely is, a realistic stance? It seems insanely hopeful to me.
Implement encryption and say the same to the FBI the last time they wanted apple to unlock something: Sorry, we can't.
It's not like encrypted cloud storage without holding the keys or client-side scanning would be new ground. Providers for that already exist and have existed for years.
I don't really agree with this system either but I, at least, appreciate the sentiment that they're trying to solve this problem while also providing E2E encryption (assuming that the internal info is correct and the end goal is full E2E encryption).
Or activists, journalists, doctors, lawyers, victims of police overreach, etc. If this is a question of framing then shouldn't apple be more than qualified to handle it?
Honest question, were are you going that has less privacy issues than this?
I _want_ to support some Linux-only phone, completely off of Google/etc. However, i suspect that will be too poor of a UX, lacking basic functionality that i deem essential. I may still buy them to support them, but i think i'm going to have to buy a Android-based phone and then heavily de-google it. I don't like this option because it still supports the Google ecosystem, but it may be a necessary evil while the Linux-phone market hopefully grows, and while i continue to buy the Linux-phone market to "do my part".
I've flashed mine 2 weeks ago and I am very pleased. Without the Google Services Framework you are restricted to the F-Droid store though (not a problem for me personally). And there are several Apps that won't work even if you install the GSF and the Aurora-Store: Banking-Apps, Online-Check-In for Flights, etc. -- again not a problem for me personally, I fly maybe once every 5 years and use Online-Banking from my browser...
HN is a weird crowd, so this obviously isn't a solution for +99% of people. But I'm choosing to opt out entirely. Living in the 90s is annoying when coordinating with unreliable people, but this isn't some kind of impossible feat. I don't even think about it anymore, it's a non-issue. Just get on with it.
They are sitting on ~$200B in cash. If they want to continue to be viewed as protecting users data and privacy, that is exactly what they should do. Spend some of that money in court, defending the privacy of their users. But they won't, and they will lose the thing that differentiates them for some subset of their users.
Yes, it's hard, cumbersome, and not "there yet" for a long time to come on a lot of fronts, but it's the only way out of the corporate multiverses.
Years ago it was something that only nerds and techy people would use and now one can easily install it for any family member and never look back.
How about not moving forward with this system?
Corporations have to obey the law, but they don't need to be zealous and build spying infrastructure when they aren't required to.
(and all of that would be before I even consider false positive rates)
But even if that was the case, the tool will now exists for any dictatorial government to spy on "troublemakers". The Chinese government, which has already gotten full compliance from Apple, would be foolish not to use it.
On one hand it's encouraging to read your take, but I have no faith that in another 10 years, the government scanning all our devices for "misinformation" that the then regime deems as such, will be anything but hand waved away by the majority of SV employees as "it's the law".
> Apple just side-stepped the question on how to reconcile trust in propietary software and surveillance requests from the government.
They actually inserted themselves into the process, not side-stepped it. They could’ve kept status quo (no scanning) and kept supplying iCloud Backups to governments at large rates. As part of this, they’re also forced to add a greatly increased human review function.
> We don't even know what big brother wants to know about you.
They’ve explicitly stated what they’re scanning for and went further to say they will not scan for other types of content. You may not believe them, but this is not a content-neutral or blind justification.
Given pressure from governments for this type of surveillance is, i think, well agreed upon - Apple giving in while avoiding direct responsibility seems quite the incentive in my view.
For sure there had to be a lot of pressure. I don't believe Apple did just because it was a beautiful morning once.
I'd guess they tried it once or twice, and it was too slow so they went for more forceful means.
This just reasoning from what they've already done in the past.
That’s a very bold claim to make on hearsay. Does your friend work at Apple? Are they just speculating, or do they have actual evidence? Why do you believe them?
C’mon, people, I know this is a knee-jerk issue, but exercise some basic critical thinking skills.
As far as the attack goes, I have a few more details, but I'd rather no reveal them since I could be violating confidentiality somehow. I understand why you'd be skeptical.
Winning!
To follow the law? You can already read a bagful of stories where it caved in on law enforcement pressure. It already follows much invasive legal systems in China or India like any other major tech company. The measures there are being presented as "anti-terrorism", maybe a less marketable term in the US currently.
Wait a second, but only if you're uploading photos to iCloud? They were presumably already doing this once you upload them, so what is the big difference? I have totally missed this detail in a lot of the coverage, and it changes my view of it.
I'm aware the general sentiment here is "don't assume governments are putting pressure on Apple to step up their reporting, this is all on Apple bad bad bad bad bad," but sorry -- those numbers do indeed lead me to strongly suspect governments are putting pressure on Apple to step up their reporting. And I don't think Jason Snell (the author of the linked article, who probably knows more about Apple's world view and internal politics than almost anyone outside the executive team does) is wrong to theorize that Apple sees this CSAM-hash-detection system as a way to meet that pressure while protecting user privacy as much as possible, because they've convinced themselves that "protecting user privacy" means "do as much on device and as little in the cloud as possible".
To be clear, I don't think this is a good idea -- letting the camel's nose in the tent through a different opening is still letting the camel's nose in the tent. The irony, though, is that I think Apple's inventively bad solution comes from the most canonically Silicon Valley mindset ever: all problems are engineering problems.
I think we have to look at this through the lens of governments around the world attempting to ban E2E encryption. E2E encryption has both security and privacy benefits, and IMO most people are far more affected by the security benefits, given the frequency of data leaks/hacks. So maybe Apple is saying "let's preserve some of the E2E encryption benefits", but show it can be compatible with LEO demands.
I think the unfortunate reality may be that that is the least bad option.
But I think your arguments would be improved by actually reading and understanding the current system Apple is proposing. Which, for example, does not include automatic reporting to authorities based only on a hash detection.
At the end of the day this is mostly an issue of political and legal requirements. Apple is doing what they think they need to do to cover their bases as they wrestle with governments over things like E2E encryption policies.
A utopia of GPL-only mobile devices with no proprietary cloud service connections doesn't seem like a valid option. At least not for a large enough share of users to make a societal difference.
Nah. Everybody else will live in that world because they are addicted to convenience. Personally, I'll be scrapping my iPhone and buying something with free software.
Apple's the first to do this on your own device. Sure, only content that would have been uploaded to iCloud. For now. But this is the start of that particular slippery slope.
Any form of automation and convenient goes against the main point that is privacy.
If you share cryptomator and the vault to your family/friends they can access it but it's not possible to look at the pics on the web that way.
The app is also paid but it's a one-time fee and not expensive.
I don't know what Apple is thinking, but I doubt it's that, because people who care about being surveilled are very unlikely to be persuaded by that argument. (And obviously people who don't care much about being surveilled are unlikely to care very much whether such surveillance happens on device or in the cloud.)
Edit: I guess at worst Apple's just sending a cryptographic hash of unencrypted data so that the known childporn can be flagged. It's basically like sending a hashdeep record of the data. It's an interesting concept with at lot of implications for tracing the spread of files within networks without knowing what they contain. Presumably if it was expanded to all files in cloud storage then LE could say something like "Give be a list of everyone that has object XDFAECXDZE" when LE knows that XDFAECXDZE is Catcher_in_the_Rye.pdf and Apple does not.
My understanding is that this only works for known abuse images and even then only if the files are identical. So apple has no way to know if high schoolers are sending nudes to each other.
No, we don't. Your refusal to understand how any of this actually works, your desperation to conflate this process with the worst hypotheticals, does more damage than Apple ever could.
I have an old razor flip phone I might start using again. That and an iPod touch with no cellular.
This is here to stay. Time to adapt to it.
This is straight up a lie. Comparing hashes is not anything crazy, nor is it an API for big brother
As implemented, I agree with Apple
That's the point. It's not an open to the public list, it's secret and controlled by few. It can contain whatever they want it to contain.
> Not how this system works.
That's absurdly naive.
Also, I'm willing to sacrifice a tiny bit of privacy to stop child porn collections.
The hashes are then sent out to for comparison to the hash table.
Right now they are only comparing to the CSAM table but they can add other later.
Also, how are the hashes sent? Can they be intercepted?
If the Apple reviewer looks at the images that were flagged and doesn’t see CSAM, they don’t report it.
If they just hash photos, and if they only use the NCMEC database and if they only look at photos being uploaded and if the government has no access, then it is not anything crazy. But by implementing the system they invite scope creep at every one of those ifs.
But anyway, media being media, not bad to waste a few minutes feeling good about oneself that one can form an opinion about such a huge and powerful corporation.
https://en.wikipedia.org/wiki/Communism
So no - not communism.
Fascism? Authoritarianism? Perhaps?
Agree we need right to repair for ordinary users and also agree we need data protection/privacy legislation.
It's an interesting argument.
My gut reaction is that end to end is still a good definition as is, that we shouldn't update it. This highlights a separate & just real issue, illustrates that encryption still interactupon multiple other layers of platform, which are themselves security issues.
But we should try to re-rally around this. There's no catchy pro-user term like "encryption" here, or "end to end". "omniversal encryption," to imply across all time and space, or "end to end, you to me". Finding the right name for this weakness, for what we want & will not accept short of: we should find this flag to fly.
Just kidding, hunt them down and send them to hell.