Bugs in our pockets: the risks of client-side scanning
arxiv.org
arxiv.org
"In a world where our personal information lies in bits carried on powerful communication and storage devices in our pockets, both technology and laws must be designed to protect our privacy and security, not intrude upon it. Robust protection requires technology and law to complement each other. Client-side scanning would gravely undermine this, making us all less safe and less secure."
After I removed all apps that were complaining about missing Google Play Services and installed alternatives for the ones that I needed like the calculator, everything was working fine. (thanks to f-droid for helping me find viable alternatives)
I read a post here about when radioactive toys were popular, back in the early 1900s. Radioactivity fell so much out of grace, that these days we prefer to die of global warming than to use nuclear power.
In a similar vein, a world where most people go back to pen and paper and watching public TV on bootlegged devices so that they can (try to) escape continuous surveillance by electronic devices is conceivable. Give it enough time and grisly precedents, and companies like Apple and Samsung will only be able to sell new devices to corporate customers.
Hal Abelson, first name there, is from the computer science classic SICP book/MIT lecture series - https://groups.csail.mit.edu/mac/classes/6.001/abelson-sussm...
don't forget Steven M. Bellovin who's on that list and who was one of the inventors of the first Encrypted Key Exchange, a precursor to the Diffie-Hellman Key Exchange.
Richard Hellman started selling jars in 1912. No remote relation apparent. (Specifying just in case of doubt, especially for those unfamiliar with the brand.)
I don't believe this is a technology problem. This is a legislative problem. We (well, some of us) have the Bill of Rights and the justice system for a reason. We need to codify who owns our data (we do) and what third parties can do with it (nothing unless we say they can, individually).
LEO can't do a damn thing unless they have probable cause. This dragnet BS needs to stop.
We are absolutely empowered to change the rules to suit us.
This is far from a given.
They are designing all of the critical chips (SoC and crypto/enclave) themselves already, TSMC only manufactures them - and if there is one company capable to verify that the chips that TSMC produce actually match the designs, it's Apple.
What I see from a historical standpoint, pre-cloud, mobile phone/computer, personal encryption etc is that anything stored be it something on paper, something in your house, safety deposit box, whatever was available to law enforcement with controls via the courts or other mechanism. It was available when there was a legal matter. Is there disagreement that legal matters should allow for full disclosure whether criminal or civil?
Is the problem that law enforcement and other state institutions through legislative channels and courts getting just too much access without legal justification?
Notional idea: could you have a key vault? Only with court order the keys are released and your devices get opened up? Even if not implementable would that work for most people?
I do get the government mass surveillance aspect and think that needs way more scrutiny and people should be vicious in their defense of themselves and society. But it also smells like we lost that battle as private companies are doing pretty well at surveilling individuals and communities.
In theory yes a court could get access to your papers but for the average person because person to person interaction produces no permanent record and what limited record is created by correspondence isn't nearly as permanent because it requires one to take the time to file it and manage a very finite space nothing anything like your present digital life would ever come to exist to be requested.
As important is the cost of such a request. Compare the risk and expense of actually spying on a large population, to the cost of tapping their phones and paying someone to listen to a million boring conversations to the cost of bugging everyone's communications at once with a box installed for that purpose.
If you consider how feasible it would be to obtain that much privacy invading mostly useless intelligence it wouldn't historically have been hard it would have been impossible. They could have flown to Mars easier than doing that much physical surveillance unless every third citizen was spying on the other 2.
Historically I suspect such requests to logically be rare outside of settling business matters where receipts and contracts would be common or wills/trusts.
If you want to justify some sort of key escrow system you shall not find justification in returning to some hypothetical state of nature that never was.
Neither mass use of unbreakable encryption nor mass surveillance enabled by court order, nor your suggestion bear any resemblance to historical reality.
Furthermore if we can't trust the government not to break its own laws what would keep them from simply ordering your keys confiscated and ordering whomever holds them not to tell you?
To expand on your comment, I can go in my file cabinet and flip through papers I have chosen to keep with data that I chose to write down, and then drop it in the trash can. It's physical and tangible and it doesn't have a record of who I called, where I went for breakfast, or what I ordered.
Our founding fathers were very clear on their intent with regards to how they viewed governments imposing on the private lives of citizens.
Yes, I don't agree with this.
> Is the problem [...] too much access without legal justification?
That is also a problem, which isn't new (it existed in the time you described in the opening of your post as well).
> Only with court order the keys are released and your devices get opened up?
This doesn't work (and we know it doesn't, there's so many times it failed already because of systemic issues with the idea), so as a hypothetical it only serves to distract from the fundamental truth that it cannot be a solution.
> I do get the government mass surveillance aspect [...] But [...] we lost that battle
With that mindset, you have indeed lost :)
Applies both to services and devices.
Think iCloud, Facetime, iMessage, Push, FindMyPhone etc.
So Apple won't leave after you once have the phone – they remain with you. And spying isn't what Apple calls it. Smart services always have a tendency to be encroaching.
On one side Apple wants to read your lips to know your wishes in advance but on the other the authorities demand access to that knowledge.
The practical objection is that you have no recourse if you are focused on. So it's a numbers game. One that we shouldn't be playing.
You say most of us. How many innocent lives are you willing to destroy in the quest to vanquish an imaginary foe?
There is the side that says "we should know everything, because we own you" and the side that says "I prefer not to be owned". There is no good-faith way to see both sides of that, that is just avoiding a hard question because it is convenient.
Ah, so because you are not inconvenienced right now it's not an issue. That's not a good way to approach a systemic injustice, except if you look at other people's suffering and can still sleep fine.
"Client side scanning" (both in general, and in the recent Apple kerfuffle) is talking about a network client, that will be talking to servers that are owned by "them." If they wish to enforce rules over what is stored on their server then to enforce that right, the only two choices are to disallow E2EE or to perform client-side scanning.
Really client-side scanning is only up for debate when E2EE is used. The Javascript that checks validity of forms before you submit them is a form of client-side scanning, but most of the time[1] nobody cares because it's data that you intend to send to the server anyways.
1: Inadvertent pastes into fields that phone-home for e.g. autocomplete can reveal otherwise private information, so "most of the time"
Stallman was very, very right.
If you wanted to store photos in icloud on a Windows machine, you'd be using the Apple icloud client. Apple has at least some control over what software they write and ship does[1]. They can break 3rd party clients almost at will, so if they choose to be hostile to 3rd party clients that control is fairly strong.
Arguing over what amount of control Apple should exercise over what software runs on a device purchased by a consumer is mostly orthogonal to arguing over what amount of control Apple should exercise over what software can connect to their servers.
1: On a general purpose machine, debuggers and emulators can influence what software does, obviously, so the control isn't absolute.
Maybe it would be nice to have a tool that would allow you to see exactly what data the client side scanner is “allowed” to “see”?
The argument over whose computer it is applies pretty similarly. My computer is not part of their infrastructure, and they should only be controlling the software on their infrastructure.
It means precisely that Apple has no technical capability to remotely access the device.
It means any (consented) Apple software update leaves behind no hooks or backdoors that enable subsequent, remotely-initiated access.
It means there verifiably exist no code paths that allow remote exfiltration of data, other than those that pass through consent dialogs. The basic distinction between legitimate OS functionality, and malware.
It's not as if Apple's marketing doesn't heavily emphasize the "private", "E2E encrypted" aspects already.
So somebody else has to do housekeeping and hand-holding. That's where apple comes in again, or better, stays for convenience sake.
And to not complicate their own day, Apple just does it the same for all.
The whole point of end-to-end encryption is that what is stored on their server is statistically uniform binary white noise. If they wish to enforce that, there are a plethora of server-side tools (like the Diehard test suite) with which to do so.
You can disagree with that (and there are many good reasons to do so) - but “it’s encrypted so it’s fine” isn’t going to convince anybody who matters.
0: https://www.hackerfactor.com/blog/index.php?/archives/929-On....
Might just throw my phone in a campfire.
Even putting aside how much of a stretch that is, how is anybody else supposed to know which service you use? It's your personal files. That nobody else should have access to them is the point.
It's not as if Apple or whomever should be providing anyone with a list of their customers, as that should cause you to not use them too. As far as I know they don't currently make their customer list public.
What would you think about someone that you were talking to that showed you something on their phone (a restaurant listing you were both thinking of going to, or something on Maps), but then a Parler notification popped up? What if they were married and Grindr or Tinder or whatever notification popped up?
Would you judge them? Would you expect many other people to judge them, even if you don’t?
Don’t get me wrong, I don’t think Apple’s products would be problematic that way. But a big reason why is because they have and likely will continue to make decisions like the one we are discussing.
If they went full end to end super privacy, then got named by the feds repeatedly in whatever the next big csam/terrorist/whatever scandal, that could change, and that is even assuming Congress people don’t join in the action, which they’ve already shown an interest in doing.
Even if Apple wasn't scanning anyone's files, >99% of their users would not be pedophiles and no one would have any reason to assume that they were.
> If they went full end to end super privacy, then got named by the feds repeatedly in whatever the next big csam/terrorist/whatever scandal
The main effect from getting put on a list like that is to gain credit with privacy-conscious people for standing up for their users. Nobody says "oh no, I better stop using my favorite full disk encryption because it hasn't got any known backdoors in it."
> that is even assuming Congress people don’t join in the action, which they’ve already shown an interest in doing.
At which point you have government action and can bring out the constitutional arguments etc.
And none of your personal lack of being a witch is going to help you when you get the reputation as the ‘weird dude that uses that child porn storage app’.
I guarantee you 90% plus of the population, once they learn what Tor is, will have the reaction to Tor. And it’s a matter of time until it gets enough press for that. Same with anything that does what you describe (proper end to end, we don’t care what you store, and we don’t have the keys so pound sand LEO).
When something is low key/under the radar, it can be a healthy witch-hunt-free zone that also isn’t filled to the brim with witches. But something at the scale Apple is at can’t, and even Parler (which didn’t start that way) got too much visibility and ended up as you describe and got shut down.
The privacy conscious folks may nod their head and know, but most folks don’t get it, and if it gets widely used, it will be attacked this way by authority figures. The ‘think of the children’ routine is used a lot because it does work on the majority of the population.
As I said, Apple isn’t there yet, and likely would never be close - because they’re going to do things like they are now to avoid the Public backlash.
And as long as they put enough of a veneer on it, the vast majority of folks won’t think twice about continuing to use them (95%+ of current customers at least). Most of the market will continue using Android which is 10x worse near as I can tell.
The ‘mistake’ they seemed to have made here is being a bit too obvious about it and not keeping up the veneer well enough.
Overall, I suspect we’ll be seeing some move back to on-prem for a non-trivial percentage of folks because of this and other things going on. That said, I’m sure cloud will continue to grow exponentially because most folks just don’t care enough to pay to do otherwise.
This only happens when you start off with zero users and having an anti-witch-hunt policy is the only thing causing you to gain users, specially because they're disproportionately witches. Because without that policy the service would still have zero users.
It doesn't apply to any service that already has a large number of non-witches or has any effective means of also attracting users who are not witches.
> I guarantee you 90% plus of the population, once they learn what Tor is, will have the reaction to Tor.
Tor has a specific marketing disadvantage because by its nature it defeats most forms of tracking and advertising, which makes it adversarial to media companies who profit from tracking and advertising. This is why all the stories are about "the dark web" and not about that thing that helps dissidents in China and Iran evade authoritarian censorship.
It's also hard to get ordinary people to use it, and thereby understand that the technology itself isn't anything nefarious, because there is a noticeable latency cost to using it that most people aren't going to like.
None of this applies to a generic hosting services making it so they can't read their customers' personal files.
> When something is low key/under the radar, it can be a healthy witch-hunt-free zone that also isn’t filled to the brim with witches. But something at the scale Apple is at can’t
This is completely the opposite. It's the things at the scale of Apple that can do it because there aren't enough witches in the world to make a userbase the size of Apple's be more than 1% witches.
It also works for everyone as long as everyone does it, because then "no witch hunts" is common practice rather than something that makes you attract a disproportionate number of witches.
> even Parler (which didn’t start that way) got too much visibility and ended up as you describe and got shut down.
Parler was clearly the app that gained users by having a policy against witch hunts. It only started gaining a significant number of users when the other platforms started carelessly banning large numbers of users for alleged transgressions that were in many cases facially absurd, but it also attracted the people who were banned because they were actually bad.
And even then, it wasn't the users who abandoned them, it was Apple and Google and Amazon. Apple is obviously not going to do that to themselves.
It’s always been the case that your neighbors or friends or employers or whatever would judge you as they saw fit for whatever they might see or find out about, or even overhear from the local loudmouth/busybody. Regardless of what was found in a court of law.
The challenge now is our ‘neighbors’ are now whichever nosy random person in the entire world that cares to look instead of just the local gossip circle, and the gossip is recorded nearly permanently and is indexed in an easy to find way decades later. :s
As to if this means people will stop being less judgmental since everyone is likely to have something shitty posted about them at some point? History or the current circumstances don’t seem to be pointing to ‘yes’ right now.
But at some point, either everyone is going to stop sharing the smell of their farts every morning, or folks are going to have to stop caring, or we’ll literally not be able to be friends with or employ anyone.
It is hardly difficult to draw a distinction between ensuring a field looks like an expected datatype and ML analysis guessing at photo content.
In fact, trying to construct an argument conflating the two pretty much immediately runs in to the fact that one is adversarial, so it only works if you studiously ignore intent.
The other distinction is, if a client-side form validator detects a problem with the input, it tells the user so they can fix it, whereas if Apple's system detects a problem with your input, it potentially tells the police without giving you a chance to delete the maliciously-generated false positive files you've unwittingly received.
What I mean is that client side form validation is typically part of a cooperative process - I want to buy something at your store, you want to sell it to me, and smart client side validation can make that faster/easier if I typo something it can catch. It is (or at least should be) mostly aimed at helping the user; your parsing for security is (should be), as you note, on the server.
Scanning photos for [CSAM, thoughtcrime memes, poor taste, whatever] doesn't make the user's life easier, is not something anyone asked to be subjected to, and potentially can lead to a very negative outcome for them.
That's the distinction I was getting at, and yes, your second point is directly relevant there.
So the core of the problem is that they want to enforce rules over what is stored on their servers, even in E2EE form. They could just allow regular E2EE, where they ignore the content that they don't know and push back against politicians who think they are entitled to all data. They chose to push back against user's privacy instead.
What if I have a file on my phone that is already encrypted before I upload it to icloud and it gets encrypted a second time? Apple would have no knowledge about its content. Would they have to scan all my other devices too?
No it isn’t. It’s my device. I get to decide what runs on it.
I have an iPhone. The Photos app keeps telling me that it's unable to upload things to iCloud because my account is full.
I never turned it on. I never intended to upload any photos to the cloud.
I haven't signed into my iCloud account for years because I don't use it. Nonetheless, iCloud has a magical way of uploading things to something that I've literally never used.
Next you'll be arguing that people using Windows should have simply turned off online logins if they didn't want their Windows computer to phone home. Bullshit, Microsoft shoves that shit down people's throats.
So your statement of "you'd ordinarily be sending them as anyways as well" is ludicrous. That's deliberately burying your head into he sand against the fact that big business sets defaults to settings that users often have no idea were set or are buried behind huge warnings against turning them off.
As an example of this, I never once opted into any kind of data sharing, set telemetry to the lowest allowed setting, and don't remember ever signing into a system-wide Microsoft account, yet when I eventually discovered deeply hidden privacy options I found that my MS account had a log of every single application I had ever used on my W10 laptop.
It implements the feature of pressing Win+Tab to see open programs, and then scrolling down to see previously open programs.
Now I get that they need to check that I have a valid license. Fair enough. Anything beyond that should be opt-in.
There's definitely no way a new version could patch your system and turn something on without your knowledge. No, there's absolutely never been a situation where some new setting has shown up and you didn't know what it does or inspected what it was set to by default. And there's absolutely no way you could have restored a backup and not had all of your settings transfer over correctly. No, there's no way you'll ever turn the setting on and forget that it's on when you plug your device into some network. And you know you will never be the victim of any malicious activity that could screw you over in some way. You've never had some app automatically connect to something that you didn't know it could even connect to. You'll never have someone else pick up your phone and take random pictures or recordings that you don't know about because those would never get automatically uploaded because, of course, you didn't turn on that setting for yourself. You'll never have to worry about your battery going low because you turned on automatic uploads and not only did your upload happen but your device also scanned your uploads too. You never use your phone for work because your work definitely pays for a new device for you to use for work.
Gosh it sure is weird hat so many people don't want client-side scanning. Scanning your device before uploading anything is just a very reasonable thing to do.
/s
People not being able to understand the devices they use is why devs have gotten us to this point. People are too uneducated to do proper back ups, so some enterprising people came up with a way to do that for you. Peeps still get it wrong. Some other asshats come along and take advantage of uneducated people, and do malicious stuff. Fuck 'em. We should just end the cloud because we as a society can't handle it or the responsibility of operating our own equipment. /s
You're wrong because at the very minimum it means that my device's battery isn't consumed for the scanning.
Further, enabling uploading won't work if the cloud account is full (as is currently the case).
Turning off connectivity prevents the upload and scanning from occurring.
> Turning off connectivity prevents the upload and scanning from occurring.
Client side scanning which happens as part of the upload, so these apply to that as well as to server-side scanning.
So your complaint is only "battery life"?
No. Way to miss the forest for the trees. Go read the whole thread.
You either misunderstand it or are deliberately being misleading about it. The only complaint that you have made which stands is battery-life.
I'm speculating here, but I wonder if part of your experience is based on the fact that you're a long time user. Features like auto-uploading to Photo Library are new, and Apple is generally decent about informing you of new features before opting in.
Brand new account setups are a different story. You're encouraged to use all of the latest/greatest stuff (and why not, current topic notwithstanding?).
Bottom line: it's extremely easy for an average user to start uploading their stuff without really realizing it.
I'm not the unsuspecting dupe that devs are targeting to get a new user tricked into something. I'm very much aware of the shenanigans devs try and pay attention to that shit from the go.
Having said that, I do read the crap and choose no where necessary. People just haphazardly pressing okay to get to new shiny almost deserve whatever they've agreed to. I say almost because these dialogs can be worded like "Vote No for Yes" kind of BS.
If you're one of the asshat devs FUCK YOU for making this a thing we even have to discuss in the first place. Edit: Royal You Devs
And yeah.. The cloud is just someone else's computer. Would you store all your stuff on your friend's computer? Well most people store everything they have on the computers of people they don't even know...
And personally, I know this won't affect me. I don't own such content. I don't use the cloud without encryption first (thanks Cryptomator). However I just hate the feeling of my own phone constantly looking over my shoulder on someone else's behalf. Is that so weird?
The more we remove privacy by tech the less we lose it by law which I now think is the much worse outcome.
How does that work if everyone expects you to communicate with them via Whatsapp and their Gmail, or even if you don't, they will happily backup all communication with you in the cloud?
On its face, that seems like a false dichotomy. Can you expand?
Generally, I see the erosion of our right against unreasonable search and seizure to be something that hurts everyone (regardless of an individual's ability to make fewer searchable spaces).
Much like advertising those of us tech savvy enough to install add blockers are subsidized by those who don't.
That's the legal justification behind law enforcement fusion centers gathering, deanonymizing, and sharing data harvested by adtech firms, device manufacturers, and service providers.
It's a gotcha with the same intellectual weight as "I know you are but what am I?"
We need legislation with consequences lethal to companiesthat violate privacy.
Something like this - Every individual variable that a company wants to obtain from a person should be consensual with no option to select all, and the data and permission should be ephemeral. At any time a person should be able to inspect, delete, or allow continued possession of private data. They should be able to allow or deny sale or transfer of the data, and any recipient of the data must confirm permissions before taking receipt. Any algorithm or software or human analysis of data must be public and transparent. A record of any decision or business logic involving private data must be kept, and that record becomes private data, subject to the same constraints.
Stealing someone's identity should have a mandatory minimum of 2 years of community service, and total loss of opt in privileges for 5 years. No free web services or social media if you fuck around with someone else's privacy. Violations result in fines, paid to the victim and more community service.
A business caught abusing private data is subjected to a fine of 5% of company net worth per day. Half of the fine goes to the regulatory bureau, half to the victims.
Law enforcement must obtain warrants specific to known individuals - no geofencing or search term fishing expeditions. Digital data is subject to the same 4th amendment protections as physical papers and property.
Leaks would mean the end of an organization. If a company can't protect private data, then it can't participate in collecting it.
I'm sure there are flaws, but the gist of this seems a good starting outline. Anything less won't solve the problems and there should still be a mechanism for consensual participation in data markets. This would nuke credit bureaus, rein in isps and big tech abuses.
How long until you do care? Will there be anyone left to care when they come for you?
What's missing from this question is the recognition the answer can very easily be "probably never". Nothing says "they" (who?) will come for you (for some reason this is always implied) and it is eminently possible the person you are questioning doesn't mind living in a fascist hellscape where their neighbours are regularly dragged off to death camps.
Of course, all of that ignores the fact that your question is histrionic. The problems you are referring to are not modern problems and comparing them is a disservice to both situations.
Develop CSS in a manner that minimizes the noted risks. Such mechanisms are a fundamental compromise, philosophically. I am skeptical that those on opposing ends of the privacy debate will find sufficient common ground to achieve responsible implementations.
Deeper concerns regarding the misprioritization of security in consumer infotech design prevent meaningful basis to realize a suitable compromise for CSS tech, anyway.
No, do not. There is no reasonable privacy preserving manner in which you can do so. Spyware is fundamentally incompatible with privacy. I don't care how many god damn whitepapers they write about their novel perceptual hash cohort-based homomorphic 0-trust TPM scanner. It's still a rat.
At present, Apple doesn't scan photos on the server, but all their competitors do, and I don't doubt for a second that they will eventually start scanning photos as well. The choice is not if, but how, and their client-side solution seems to me to be much more privacy-preserving than server-side scanning.
¹If you don't believe them, that's fine, but given that they have root control over the software running on your phone, your only choice is to either believe them or don't use their phones. Same goes for all other phones.
For example, a full e2e encryption system where only the user owns the keys and nothing is scanned.
This is already possible today with any general purpose computer.
No, it isn’t a choice at all. Your statement is factually incorrect, and presents a false situation. Apple has no obligation, legal or otherwise, to perform CSS. Nothing is stopping Apple from allowing E2EE right now.
Of course, the same is true of messaging. Apple owns the keyboard software and nothing stops them from putting a keylogger in to capture text before it ever hits the messaging app that encrypts it (and it pretty much needs to have one for predictive text to be possible).
They obviously can and arguably should choose to ignore the data streams before they hit the network clients and can be encrypted, but the capability will always be there.
I think part of the issue here is Apple owning all of the hardware, the OS, and the network client. People don't want to trust network clients, but you have no choice but to trust the OS and hardware vendors. If you don't trust them, your only option for guaranteed private communication is to not use computers. You either need to resort to the organized crime/terrorist model of using hand carry via couriers who credibly believe you'll kill them if they rat you out, or the military model of building your own communications devices.
The fact that they don’t support E2EE, despite their strong pro-privacy stance, supports my presumption. So it comes back to the same argument: presuming Apple is going to scan cloud images for CSAM (which, again, all other major provides already do, to my knowledge), then it’s just a question of how.
For someone like me, who believes scanning for CSAM is worthwhile, Apple’s solution is far superior and privacy-preserving compared to, say, Microsoft’s.
I think this argument really comes down to “no scanning at all” vs. “carefully applied scanning,” but that’s not how it’s framed by the people objecting. I think it’s because that’s an argument they’re not likely to win. And so, if they “win,” I think we’ll just end up with cheap and dumb server-side scanning, which would take a whole lot less effort and political trouble for Apple… and ironically, be much easier to subvert in the ways people against CSS worry about.