This all plays so perfectly into my long-time fears about the locked down nature of the iPhone. Sure, it's more secure if Apple is a good steward, but no one outside of Apple can inspect or audit what it's doing!
This all plays so perfectly into my long-time fears about the locked down nature of the iPhone. Sure, it's more secure if Apple is a good steward, but no one outside of Apple can inspect or audit what it's doing!
In all these threads everyone is coming close to the crux of the issue, but I want to restate it in clearer terms:
There is a sacrosanct line between "public" and "private," "mine" and "yours." That line cannot be crossed by Western governments without a warrant. Cloud computing has deliberately blurred this line over time. This on-device scanning implementation blows right past that line.
Our tools before the computing revolution, and our devices after, become a part of us. Our proprioception extends to include them as part of "self." A personal device -- a tool that should be wholly owned and wholly dependable, like pen and paper -- that betrays its user, is a self that betrays itself.
And cloud computing are corporations that take your self away from you.
And that was always going to happen. And people have been fighting that idea in the shadows for a while.
This issue is just bringing it to the foreground in a most intense way. "Think of the children."
They are coming for your self. They will take it. There's not doubt. There never was. Your self is too valuable and you've just been squandering it anyway.
Eventually you become Apple. Unless you start resisting, but that's hard. Just slooooowly warm the water. This froggy is fine. Nice and warm. Just don't touch the bottom of the pot.
This is a self-delusion, I am afraid. The line has been crossed more than once, and it will be crossed again. UK and Australian governments are just two prime examples of waving terrorism and pedobear banners as a pretext to get invasive with each new legislation, and the Oz government already has a new legislation draft to make it a crime to refuse cooperation with law enforcing services when they request access to the encrypted content (think Signal messages). Also, refer to the Witness K case to see how cases that are unfavourable to the standing government completely bypass a «trustworthy» Western judicial system, including the Minister of Justice.
CSAM is guaranteed to be abused under whatever new pretext politicians can come up with, and we won't even know that Apple has been quietly subjugated to comply with it in those jurisdictions. There will be even less transparency and even more abuse of CSAM in «non-Western» countries. That is the actual worry.
Sir James Wilson Vincent Savile OBE KCSG (/ˈsævɪl/; 31 October 1926 – 29 October 2011) was an English DJ, television and radio personality who hosted BBC shows including Top of the Pops and Jim'll Fix It. He raised an estimated £40 million for charities and, during his lifetime, was widely praised for his personal qualities and as a fund-raiser. After his death, hundreds of allegations of sexual abuse were made against him, leading the police to conclude that Savile had been a predatory sex offender—possibly one of Britain's most prolific. There had been allegations during his lifetime, but they were dismissed and accusers ignored or disbelieved; Savile took legal action against some accusers.
Asking the police to put some investigative effort in rather than a dragnet sweep of every photo on every persons device is not too much to ask.
Importantly this should be the hard-capped precedent for every sort of crime. There's always a ton of other ways these people get caught without handing over preemptive access to every innocent persons phone. Same with terrorism, murder, etc.
I see push for CSAM as an alignment of powers in the form of "The Baptist and Bootlegger." The well meaning moral campaigners are a tool for folks who seek to surveil and control.
Even if the cops and private security along with TLAs are actively spying on people on an extremely regular basis, we're better off if officially they aren't supposed to be, if some court cases can get tossed for this, that they have to retreat sometimes, etc.
This is why having Apple overtly spying on the "private side" is still a big thing even with all the other revelations.
The fact that the line is there gives us a reason and a foothold to fight back when someone attempts crossing it.
I prefer to think of it as being just one national security letter away from that happening.
Which is of course a schroedinger's cat kinda thing. It's already been sent. Or it hasn't. But why would the national security apparatus not take advantage of this obvious opportunity?
Anyone giving benefit of the doubt to this kind of stuff nowadays is IMO very naive or just poorly informed of recent digital history.
For a normal law enforcement context, say you bust 1 drug dealer, and are trying to find more. Maybe they have an image of a "contract" (document outlining terms of a drug deal) on their phone. You could find the other parties to that contract by searching everyone's phones for that image.
For a national security context, you could imagine equivalents to the above, you could also imagine searching for classified documents that are believed to have been leaked, or maybe searching for documents you stole from an adversary that you believe there spies are likely to be storing on their phones.
I'm saying documents here instead of images, but plenty of documents are just images, and I have little doubt that they could get this program to expand to include "scan PDFs, because you could make a CSAM into a pdf" (if it doesn't already?).
Then for version 2.0, they'll realize the INTERPOL, FBI, whatever can provide them a combo hash plus one string of text that can nail a known active producer. The holy grail. This small feature add to get so much closer to the original "moral goal" will prove too appealing to pass up. Now all the code is in place for the govt to pass over requests for data pinpointing mere common criminals.
Outside the US and other western countries, this could be a government (Russia? Chechnya?) forcing Apple to look for LGBT pride flags.
Just make it do equivalent of .* of all photos on device. It would be hard to argue, that's more difficult than scanning for specific hash.
And there is nothing specific about images. Extending this to scan arbitrary data is probably not that much code, depending on how its program maybe configurable.
The only difference is intent. The technical risk has not changed at all.
And not just Apple. Once technical feasibility is proven, I can easily see governments mandating this scheme for all devices sold. At that point, it can get even more ugly, since e.g. custom ROMs and such could be seen as a loophole, and cracked down upon.
I will also note that, while it may be a hypothetical in this particular instance as yet, EU already went from passing a law that allows companies to do something similar voluntarily (previously, they'd be running afoul of privacy regulations), to a proposed bill making it mandatory - in less than a year's time. I don't see why US would be any different in that regard.
At this point of the discussion, people usually pivot to scope creep: the on-device scanning could scan all your device data, instead of just the data you put on the cloud. This claim assumes that legislators are too dumb to connect the fact that if their phone can search for dogs with “on-device processing,” then it could also search for contraband. I doubt it. And even if they are, the national security apparatus will surely discover this argument for them, aided by the Andurils and NSOs of the world.
As I have repeatedly said: the reaction to this announcement sounds more like a collective reckoning of where we are as humans and not any particular new risk introduced by Apple. In the Apple vs. FBI letter, Tim urged us to have a discussion about encryption, when we want it, why we want it, and to what extent we should protect it. Instead, we elected Trump.
The precedent now established by Apple is that it's okay to scan things that are physically in possession of the user. Furthermore, they claim that they can do it without actually violating privacy (which is false, given that there's a manual verification step).
Apple’s claim, further, is that this approach is more privacy-preserving than one which requires your cloud provider to run undisclosed algorithms on your plaintext photo library. They don’t say this is not “violating privacy,” nor would that be a well-defined claim without a lot of additional nuance.
Instead, a forum of presumably technically savvy people are reduced to hysterics over implausible futures and a letter to ask Apple to roll back a change that is barely different from, and arguably better than, the status quo.
We need both - develop free software alternatives (which means to stop pretending the alternatives are good enough), and to get real about supporting legal and governance principles that would protect against abuses.
If people want to do something about this, these are the only protections.
icloud scans for those faces
finds those faces and ties them to other ID accounts via face - then via fingerprint recognition to a device, and to a location based on IMEI etc.
Apple's platform is literally the foundation for the most dystopian digital tool-set in history...
Once the government is able to crack the apple war chest, everything is fucked.
The "one in trillion" chance of false positives is Apple's invention. They haven't scanned trillions of photos and it's a guess. And you need multiple false positives, yet no one says how many, so it could be a low number. Either way, even with how small the chance of it being wrong is, the consequences for the individual are catastrophic. No one sane should accept that kind of risk/reward ratio.
"Oh, and one more thing, and we think you'll love it. You can back up your entire camera roll for just $10 a month and a really infinitesimally minuscule chance that you and your family will be completely disgraced in the public eye, and you'll get raped and murdered in prison for nothing."
So iCloud Photos circa 2020 [and Google Photos and Facebook and Dropbox and OneDrive] aren’t a risk you should be willing to take.
This feature doesn’t change anything in that regard; the scanning was already happening.
Apple's new policy is: "of course your phone is scanning and flagging your private files to our server - that's normal behavior! Don't worry about it".
Debian tries to be reproducible, so to avoid being caught they might need to control the mirror to so that they could send it to only me. I.e. if I'm lucky it would take a total of 2 people to put malicious binaries on my computer (1 with a signing key, 1 with access to the mirror I download things from).
It’s really disingenuous to suggest that open source isn’t dependent on trust, you just change who you are trusting. Even if the case is someone else is auditing that code, you’re trusting that person instead of the repository owners.
I’ll concede that at least that possibility to audit exists but personally I do have to trust to a certain extent that third parties aren’t trying to fuck me over.
With Apple you have to trust them and trust they don't get national security order.
I trust that if everyone who had the ability to audit OSS got a national security order it would leak and it would be impossible for many who live in other nations.
Suppose Debian's dev process happened at monthly in-person meetings where minutes were taken and a new snapshot of the OS (without any specific attribution) released.
If that were the case, I'd rankly speculate that Debian devs would have misrepresented what happened in the openssl debacle. A claim would have been made that some openssl dev was present and signed off on the change. That dev would have then made a counterclaim that regular procedure wasn't followed, to which another dev would claim it was the openssl representative's responsibility to call for a review of relevant changes in the breakout session of day three before the second vote for the fourth day's schedule of changes to be finalized.
Instead, there is a very public history of events that led up to the debacle that anyone can consult. That distinction is important-- it means that once trust is in question, anyone-- including me-- can pile on and view a museum of the debacle to determine exactly how awful Debian's policy was wrt security-related changes.
There is no such museum for proprietary software, and that is a big deal.
Most people don't care to follow the controversies of tech unless it becomes a tremendously big issue, but even then, as we've seen here, there are plenty of people that simply don't have the technical acumen to really do any meaningful analysis of what's being presented to them and are depending on others to form their opinion, whether that be a friend/family member or some tech pundit writing an article on a major news organization's website.
Trusting Apple presents a risk to consumers but I'd argue that for many consumers, this has been a reasonable risk to take to date. This recent announcement is changing that risk factor significantly, though in the end it may still end up being a worthwhile one for a lot of people. Open Source isn't the be all end all solution to this, as great as that'd be.
For regular Linux distribution there are maintainers updating packages from upstream source that can spot malicious changes slipped in upstream. And if maintainers in one district don't notice, it is likely some in onether distro will.
And there are LTS/enterprise distros where upstream changes take much longer to get in and the distro does not change much after release. Making it even less likely a sudden malicious change will get in unnoticed.
But, I suppose my iPhone was never really mine either. I knew that, I just never quite put two and two together...
So you are saying you have nothing to fear because you aren't hiding anything?
Is this a tacit admission that you don't want them scanning your phone for CSAM because they will find something?
I'm obviously not seriously accusing you of anything, just pointing out how your line of argument applies equally to privacy whether on the cloud or on your device.
No, I'm saying that if you're hiding something, maybe don't give it to someone else.
People here too often only think of these issues from the perspective of the type of person who browses HN. Apple is thinking of this from the perspective of an average iPhone users.
There is a huge separation in the importance that the tech community puts on the general concept of privacy while the average person rarely worries about it.
if (willBeUploaded) {
scanPhoto();
}
can become if (true) {
scanPhoto();
}
Obviously, this is stupidly oversimplified, I have no idea how Apple has structured their code. But the fact of the matter is, if the scanning routine is already on the phone, and the photos are on the phone, all anyone has to do is change which photos get scanned by the routine...Photos for your own use that you do not upload to Apple’s cloud are completely unaffected.
But why are people only objecting now when they were already doing on-device analysis?
CSAM detection is a complete red herring. It is less of a general purpose scanning mechanism than the search function that was already there.
If a bit can be flipped to make the CSAM detector go evil, surely it can be flipped to make photo search go evil, or spotlight start reporting files that match keywords for that matter.
There is nothing special about this CSAM detector except that it’s narrower and harder to repurpose than most of the rest of the system.
As your immediate parent has already said: there isn't any process by which an image of a hotdog will be sent to someone else without my knowledge.
Neither "photo search" nor "spotlight" report anything to third parties currently.
On the other hand, making the CSAM code scan your text documents would be pointlessly difficult given that the other mechanisms exist.
It has already been added. That's the point here. The phone already has the code to report its own owner. People are rightfully pissed about that.
Also, adding a one-liner of code is a lot more than a bit-flip. That's the addition of a feature, which is a major business decision. Code that already exists is a lot closer to "misfiring" than code that simply does not. Flipping a policy config, OTOH, could be "explained" away a lot easier. In fact, Big Tech does it all the time! (Remember Facebook's "Oh. Sorry. It was a bug."?)
----
Please stop this poor attempt at moving the goalposts by saying, "But, similar code could be added at any time in the future! Why're you guys complaining now?!"
It's also not about where the scanning is happening, or whether that's new or old. The old scanning is not a problem, precisely because it does no reporting. The new scanning IS a problem, precisely because it does the reporting.
The former is not a problem on its own. The latter is. That's the point you keep missing.
This whole ‘now that they have a scanner it’s easier’ reasoning doesn’t make sense. iOS already had numerous better matching algorithms if your goal is to do general spying.
It’s way more complex than that, and has been engineered to prevent exactly that scenario.
I recommend you actually check out some of Apple’s material on this.
The idea that it’s just a list or fingerprints or hashes that can easily be repurposed is simply wrong, but is the root of almost all of the complaints.
I feel like you may not have actually read the paper in depth. The paper clearly shows this is a generalised solution for detecting content that perceptually matches a database. The "what" is CSAM today, but nothing about the technicals require it to be CSAM.
To answer your specific response of safety voucher and visual derivative:
- The 'safety voucher' is an encrypted packet of information containing a part of the user's decryption keys, as well as a low resolution, grayscale version of the image (visual derivative). It is better described as "backdoor voucher".
- The 'visual derivative' is just the image but smaller and grayscale.
None of those two technologies have anything to do with CSAM. Apple's statement that they will only add CSAM to the database is a policy statement. Apple can literally change it overnight to detect copyrighted content, or Snowden documents, or whatever.
> Apple can literally change it overnight to detect copyrighted content, or Snowden documents, or whatever.
No, Apple doesn’t run the database. NCMEC does.
Also, in the US, the 4th Amendment protects against the abuse you are talking about:
https://www.economist.com/united-states/2021/08/12/a-38-year...
It can target a pride flag just as easily as CP.