On Apple’s “Expanded Protections for Children” – A Personal Story
areoform.wordpress.com
areoform.wordpress.com
The very fact of existence of such Spyware Engine has effect that each person would behave as 'someone is looking'.
This means zero privacy. This means 'no privacy' feeling. Everyone is more 'careful' when people around. And this has prolonged effect on society's freedom, democracy and rights of individual because independent thinking happens when true privacy is there.
Even a fear of false triggering the system and exposure of one's private thoughts/pictures/events to other people during 'sorting out false alarm' would make people behave differently. Even if they have nothing to do with any children the fear that their private moments can become public just because some AI would decide so would make each person think twice about each step.
This is a huge attack on privacy of individual and should not be taken lightly. The effect will be also huge. Just imagine what they will do next if they get away with this. Just imagine what other companies will do once this is accepted for apple devices.
I think this can/should also be considered as a fraud because when one bought apple device one was never told that spyware would be installed some day later.
Edit: I think it should be illegal/made illegal to install any spyware on any device for any excuse without warant. Spyware is a Search of 'home' without warrant to some degree, isn't it? Done through the hands of some private company under some excuse... What about Fourth Amendment?
And it is always pretended that the bad guys can only exist on the client side and not on the supervisor side.
For instance, let's imagine a technology emerges that can render any wall in a building transparent and penetrable for a short time, without affecting the structural integrity of the building. How much easier would it make the job of the law enforcement! How many crimes would it help prevent or at least detect! Of course it will be a securely guarded technology, so that it won't fall to hands of random strangers, malicious hackers, or burglars; only the law enforcement agencies would use it, and only for legitimate purposes! Honest.
Would you endorse such a technology in your town? Mandate it for your neighborhood? Why? Are you comfortable with the idea that a number of crimes will go undetected or not prevented because this feature is not implemented? Not rhetorical questions all.
> Destroying the privacy of several billion people is not an adequate price to pay for capturing a dozen or even a hundred bad guys.
> Sure it did get them some. So would carpet-bombing New York City. Success alone is a worthless measure without taking cost into account.
-- http://yro.slashdot.org/comments.pl?sid=4631081&cid=45871537
Or
http://www.usatoday.com/story/news/2015/01/19/police-radar-s...
This is stuff directly from Deus Ex (1999). Repercussions for posting wrong memes are also mentioned in Deus Ex, and are also coming true. (The epidemic situation is not that dire as depicted there, though. At least not yet.)
* Say bad things about the king.
* Said good things about the king.
* Are homosexual and attempting to live their life.
* Are black and attempting to live their life.
* Are foreign, regardless of activity.
* Are female and attempting to own property.
* Are local but not the right type of local.
* The actual person is OK but they're trying to help the Jews.
* Own the wrong book.
* Worship the wrong god.
And in hindsight it is generally agreed that those laws were poorly thought out. Giving the police tools to enforce 100% compliance with the law is by no means a sane thing to do. And if that is the plan it would behove us to make sure the law is good first. Which it obviously isn't there are gaping holes in every legal system.
https://www.sciencealert.com/wi-fi-signals-can-identify-you-...
https://www.nbcnews.com/tech/tech-news/mit-device-can-detect...
Not sure I see the relation to Apple laying Fully Automated AI ThinkPol groundwork though.
Let's assume that it only works with thin walls, so it has limited archaeology use. Firefighting is a very fair point.
Would you, reader, trust the firefighters to not abuse such a power by mistake? Would you trust the chance of it being abused for the chance of being rescued during a serious fire? Not an easy question.
Here lies the problem. You can't guarantee that. Law enforcement itself has criminals. In some countries law enforcement is even one of the largest collection of criminals. And then something like Trump happens. Do you really want to give such powers to people like Trump and it's followers, not to mention more malicious ones?
There is no way that this technology is only used for good.
And what comes on top, it doesn't stop crime, just it's modus operandi.
If they can't use their computer to handle certain data, they use hacked ones and hide it there. You know the trick how drug smugglers use the suitcases of tourists to smuggle their drugs?
I run a shipping company. You want to send a package. It is illegal for me to handle and ship certain things (e.g., nuclear bombs). Further, I don’t want to handle any of those things. Furthermore, if I find those things (e.g., when a package breaks open), I am legally required to alert the authorities.
What assurance can you provide to the me (the shipping company) that there is nothing illegal in your box? Suppose I ask you to attest but then I find out later that a bunch of people have been lying on their attestation forms which means I have been unknowingly, undesirably made party to illegal activities? Every other company solves this by simply opening everyone’s package and looking. Suppose my shipping company’s clever engineers invent a detector I can give to you that doesn’t require me to look inside the package but can tell me with some certainty that there are no illegal things in the package. What statistical properties would the detector need to have to satisfy you that this was better than forcing every package open?
We already have real world evidence for “what statistical properties the detector would need to have” to be better than opening every package, because the real delivery companies are literally doing this today. There’s nothing hypothetical about the question.
Apple could do the same, simply letting the user know that this/these images cannot be uploaded to iCloud, and then do nothing else.
The problem is that the results of these scans are pretty useless. They don’t prove anything. While Apple knows that law enforcement and politicians will believe it’s 100% correct, because they don’t even understand that DNA can be wrong, and demand customer names from Apple.
This is very much the crux of it.
Back during the immediate post-9/11 era there was a huge push to restrict all kinds of civil liberties (and the birth of today's surveillance state) in the name of preventing the next terror attack, which could be "a mushroom cloud" in the words of George W. Bush.
There's a problem with this reasoning.
Osama bin Laden wasn't some random dude who got radicalized. He was a member of one of the wealthiest families in the world. He was born rich, grew up rich, attended Harvard, and had a family that rubbed shoulders with heads of state and were directly connected to the Saudi royal family.
Osama bin Laden was a member of what I've heard described as the "superclass," those who are beyond just being "merely rich" in that they possess not only vast wealth but internationally diversified wealth and powerful political connections.
He was of the social class that is behind the glass of a surveillance state, and if he wanted to avoid any possibility of surveillance himself he and other members of his class could easily afford expensive security consultants and specialized devices. They could also afford armies of attorneys to get them off any lists and out of any trouble.
If someone detonates an atomic bomb in Washington DC, it will not be some random middle class youth who got radicalized on a 'chan board or a Facebook group. It will not be some random protestor or dissident. It will be someone like Osama bin Laden. It will be someone with the money, expertise, connections, and background to find, recruit, and pay the personnel required to obtain or build a nuclear device. It will be someone with the connections to organize the logistics to smuggle it into the country and put it in position.
It will be a member of the global elite.
People with bin Laden's level of wealth and privilege are the dangerous ones. The lens of scrutiny should be aimed at them. One member of the middle class radicalized with toxic ideology might shoot up a school, but one elite radicalized with the same ideology could blow up a city or engineer a super-plague.
Osama bin Laden's $30MM (per Wikipedia) inheritance from his estranged family no doubt greased some skids, but mission-driven people with charisma come from all socioeconomic backgrounds, and some of them succeed. (FWIW he did not attend Harvard, and AFAIR never set foot in the US.)
The general pattern is that the wealthy are, on average, supportive of the existing order. They are the winners of the current game. Of course there are always rebellious children who are motivated by religion or power or fame etc, but most of them are feckless due to their upbringing. And ultimately, there are so few of them.
Leadership abilities can be found across all economic strata. Most are happy to leverage their talents into moderate economic advantage, but some are driven by "larger" causes.
I would say that the intersection of leadership abilities, belief in a "larger" cause, belief in victimization, and a willingness to harm innocent people (or to blame them for their inaction against your oppressor) ... is what leads to the risk of violence against the existing order.
https://en.m.wikipedia.org/wiki/Osama_bin_Laden
My point is that people with money and power are far more able to execute large scale crimes, and not just terrorism. Meanwhile surveillance is experienced more and more as you move down the socioeconomic pyramid.
Apple is catching flak for this but overall their devices respect privacy more than most cheaper devices. They also cost a lot more. Your average cheapo phone or laptop comes absolutely stuffed to the gills with spyware and runs older OSes with bad security. The poorer you are, the more spyware riddled and insecure your devices probably are.
Do not discount the effects of chan boards and Facebook groups so foolishly.
Some of it may have been organic at first, but things don't stay organic for long these days. The instant there's even a whiff of a popular movement that can be exploited the propagandists are on it... especially if it's a movement that can be exploited so as to win an election or make money. The thing that made that stuff dangerous was elites and their water-carriers like Steve Bannon, Donald Trump, Milo Yiannopolis, Rupert Murdoch, etc. empowering and steering all those useful idiots.
The Pajama Nazis have a camp of doppelgängers that I've come to call Basement Bolsheviks, but they haven't had much impact since that camp of idiots doesn't seem useful to anyone with money and power (yet?).
Apple can decrypt data on iCloud, so why not just do this offline? Doesn't make much sense.
It makes me think that the technology will be expanded to encompass all offline images at some point.
However... yeah. This is a massive breach of trust. Having any spyware agent on my device is disgusting. I can't imagine the abuse cases this sort of thing can be used for.
https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
However there is a very long list of companies not just doing things for good, but also for profit.
And when profit enters the equation, then the original mission statement tends to get distorted into something bad/worse.
Of course Apple is free to decide, but the consumer has the option to either loudly disagree or exit the Apple ecosystem altogether.
I see the same pattern with pro-vaccine and "anti-vax", either you support vaccines or you are labeled "anti-vax", all is black-and-white without any nuances.
That sort of discussion rarely leads to any productive outcome.
I have an itch that somewhere in the hundreds of pages of EULA no one ever reads there is a "You accept that Apple has the right to modify its software at any time for any reason" kind of thing
You have no idea the inherent power of being the "setter of definitions" in a legal context.
The presence of legitimate common sense (an actual concordance of principles, life experience, understanding, and the existence of a fungible nominative signpost that will be reliably reproduced within error bars) is surprisingly difficult to hew and maintain. It morphs over time, and is exactly why these multi-national tech companies implementing things like this is so terrifyingly dangerous. If they implement the capability to do a thing, they are now the movers of the political Overton window by realizing the means. They are literally shaping the rhetorical landscape.
Example: Let's say the push for this intrusive client side scanning came from the U.K., and the U.K. marketplace alone accounted for enough business to make it too painful to NOT do (simplifying economic/business assumption for illustrative purposes, and just picking on the U.K for because they are $not_my_jurisdiction). In the U.S. (Third Party Doctrine aside), this feature would be something that would be a U.S. Constitutional violation (4th Amendment) if mandated by the legislature short of an Amendment, yet since it would be done out of expedience in the absence of someone telling them NOT to do it, they'd ship it by default on American phones anyway. The mere existence of the capability greatly increases the willingness of jurisdictions to use it. This means the most privacy eroding jurisdictions are creating a race to the bottom for infecting everyone else barring population Al refusal to say No unambiguously.
I don't know why this jurisdictional backdooring isn't more obvious to people than it is. Or maybe I'm just starting to grasp how politics actually works vs. working in theory, but this really does seem to be completely screwing up how process is supposed to go. This is nothing short of a private entity taking the practical reins of power, and the political edifice coming along behind and post-facto rationalizing what the actual trailblazers are doing.
To be frank, this terrifies me even more than Congress being in charge. Congress's process rounds out most things to some level of effectual benign nature. Tech companies though? If you even start asking the important questions about higher order effects, either out the window you go, or everyone looks at you like a nut.
This is not a good way to go.
If Spyware Engine is allowed to be installed by some idiot with company in his hands to amplify his idiocy, what prevents another idiot with company in his hands to do the same?
And how about specially crafted 'normal' pictures sent to individual to trigger the system? Special mind would have a perfect tool to get some one down / put into the troubles.
Exactly this was my first thought. Sad that these days my first thought, if some new "system" to prevent/catch something/one is introduced, is "How could this system be abused to blame somebody?".
I don’t think that’s true. Compare the difference in how people react to cctv vs how they react to “Surveillance Camera Man”.
This is a cctv system, you can forget it’s there. Until it incorrectly flags you anyway.
People are aware where and how they are recorded.
Phones are (well.. were) private, people take nudes, do naked video calls and all the other related stuff, because they trust that there is only one other person watching this, and they usually trust that person.
Now, we're one step away from trump-beats-cnn-gif-meme getting into the hash database to flag all the trump supporters. When the next wikileaks happens, and they can track who had the documents/photos/videos first, possibly even before they got leaked. Someone (4chan,...) can email you photos an videos that are of something random, but crafted in a way to create a false positive (a bit slower form of swatting).
This is basically setting up cctv in your bedroom, saying it's there "to protect the children" and trusting some stupid algorithm to look at you.
She gave it to me to get back in working order, and I had to ask her if she was interested in pressing charges, because handing it to me may compromise chain of custody of any investigation. She decided not to proceed with filing charges, but just wanted me to investigate and restore the bloody thing.
The person in question sync's their mobile content to that laptop. I had more than enough evidence of phone use while driving, who they were associating with, where they were hanging out, what they were doing, etc.... I didn't even have to try digging.
Do not underestimate the level of mindless information spillage by modern devices, nor the level of exposure guaranteed always on, officially acknowledged hooks into your device sensor or input feeds offers.
Even if not illegal in any way, it’s one of those things where even an accusation can be seriously damaging. The prospect of your phone continually scanning your photos will make that even worse.
They (that is, we) should already. We should assume being looked at, unless we can reasonably prove it's not so. In all public places, for certain. Using all public services, for certain.
This decision erodes the trust in personal handheld terminals (quaintly still named "phones"). This is great that the announcement is made publicly. I can easily imagine a similar feature to be deployed tacitly in other countries and other platforms.
No need to imagine.
China flat out censors chat programs and discussion sites, and some words will instantly get you put on "the list".
Winnie the Pooh, for example.
Oh, you say that you're just interested in the wonderful literary works of A. A. Milne, but that's what the anti-government types always say! Prove your innocence in this kangaroo court, and you'll be let off with merely a stint in a re-education camp.
I can’t say sexy. In a dating app. With a woman I matched with. What am I supposed to do? Say she’s doing a great job of maximizing her gene pool’s potential for aesthetic presentation in the context of a romantic setting?
If she goes for that, she's a keeper.
Would you go into a Christian church and start preaching about Hindu gods? Some might, but nearly everyone would say it's disrespectful of a different culture.
So it seems to me like it'd be a cultural difference. In Western culture sex is prominent (America, Europe, etc) even if repressed (America). In Chinese culture... well I'm going to assume it's still very much repressed.
But the point about cultural differences does stand (not that it should result in a government being able to police how two adults communicate.)
Two adults can negotiate their own communications protocol. Just because you met someone in a Catholic church, doesn't mean they aren't interested in Hindu gods or in sexuality. People are not defined by the culture they live in.
A dating app that tells you what you can or cannot say in a 1:1 conversation isn't a conduit for private communication - it's acting like a chaperone on the date. Except normally, chaperones are there to prevent sex, not police thoughts.
I agree.
> Simply chalking up social control to “cultural differences” ignores the fact that these aren’t cultural mores developed independently, but the results of a highly moralizing tyrannical government trying to assert social control
I've never been to China. But even from an outsider's point of view your statement strikes me as self-centered. What right do you have to enforce your own morals upon an entire other nation?
> a highly moralizing tyrannical government trying to assert social control for the material benefit of those at the top.
Western countries are arguably the same. How does that make the Chinese government wrong and Western countries right?
> Seems like you’re exercising a lot of social control over the couple.
To circle back to this comment; am I? The control is over the internet. Until they've met in real life you're just a stranger on it. Strangers on the internet can be very dangerous and with very little repercussion. Governments have an obligation to protect their citizens. So you should speak kindly and respectfully. You never know what words might upset the other end.
When you meet in person then you're putting your own person at risk for the words you say instead of just an online pseudonym. When you meet in person the stakes have been raised. What does the government do to prevent the couple from talking sexy after they've met in real life? I'm sure there's a lot but once you've met in real life then either you're physically in their culture or they're physically in your culture and respect boundaries must have changed.
Can that system be abused for the people at the top? Absolutely. Is it abused for the people at the top? Well I live in America and so I see a lot of anti-China propaganda so I'd argue that it probably is. But you'd be blind to think that China's alone in that.
Careful with this argument. What right did the Northern States have to enforce their own morals upon the Southern States (which had ceded, and were therefore their own country)?
I'm not saying you're wrong. I'm saying this is not valid justification.
That's a very good argument. But let me counter: we fought a war over it. That's what gives us the right to enforce the North's morals over the South's. Do you want to fight a war against China over our differences?
Even disregarding the war itself; that was about slavery and human rights (freedom) with strong undertones of racism. So if you want to make privacy a human rights issue then sure and yes. I'll even agree with you: I think privacy should be a human right! But until privacy is a human right recognized and enforced then again: what right do we have to enforce your own morals upon an entire nation?
UN's UDHR Article 12 [0] is very light about privacy (and what it even means) and (IMO) has a very very poor history of enforcement of human rights. Even more, it states:
> No one shall be subjected ... to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.
And I would argue that statement extends to calling someone "sexy" which might be an insult to someone with different morals than yours.
[0]: https://www.un.org/en/about-us/universal-declaration-of-huma...
Just like a porn website doesn't shield you from porn because it's "on the internet" , a dating website shouldn't shield you from intimate language.
Aren't they suggesting that the individuals involved decide what kind of language is appropriate for their conversation? I can't imagine how you'd call that "enforc[ing] your own morals" upon anyone, let alone an entire nation!
s/sexy/hot
The US has its own problems, but who should take something like the EU seriously? For what stands this union of countries? For absolutely nothing. Extremely weak, especially considering the union was advertised as standing for common values.
I’m always surprised when people say things like this.
As if we haven’t been carrying around highly capable “Spyware Engines” this whole time.
Apple doesn’t need the cover of stamping out child porn to spy on you. They can do it just fine, if they want to, without it.
I think Apple checking photos before they uploaded to its own servers is their way around the Fourth Amendment while using relatively similar methods that they already use for malware.
Devil's advocacy aside, I buy the slippery slope argument and I find these methods reprehensible and open to abuse. I agree with all of it. I think this new addition is a step on the downward slope. I don't use MacOS, nor Windows, nor any cloud services personally. I keep an iPhone 7 around to chat with "iMessage" friends -- and this is the straw that stops me from using for that. I have never used iCloud for photos even when I was using an iPhone regularly.
I used to have an Android phone, which I also never sync'd with Google's cloud/drive. Then after an update, the sync-to-cloud option was automagically turned on again and my photos started to upload. I deleted them from the cloud and swore off ever using an Android phone again.
[1]: https://support.mozilla.org/en-US/kb/how-does-phishing-and-m...
Addendum: That Mozilla support link is also interesting in terms of privacy. It's flagged as a privileged page for me so it loads Google Analytics that would otherwise be blocked by browser extensions.
> The issue here has nothing to do with children, they would always use 'children' as excuse. The issue here is installation of Spyware Engine.
I'm not sure that's entirely true. There is a reason stuff like this is debatable and is being considered by a reputable company. It is not black and white and both sides have some valid arguments. There are absolute atrocious monsters in this world and apple is in a tough spot. They are locking down their devices and it is keeping both bad and good guys out. You may think good riddens, they both shouldn't be able to get in but there are downsides to building a completely impenetrable device (e.g. tor is great for freedom, but also makes it a lot easier for criminals to operate). They are receiving pressure from law enforcement and no doubt people in their company have glimpsed the horror of child abuse and want to do something about it. They wouldn't be able to sleep at night if they did nothing, but they also won't be able to sleep at night because of the incredibly slippery slope they just stepped out on.
We also don't live in a libertarian society and I don't think most people would like the reality of living in one. On the state security and personal liberty spectrum, I think they united states is a lot closer to libertarian than we are to the ccp tho but there is no spot on that spectrum where everyone will be happy. Finding the least bad spot is an incredibly difficult problem for Apple.
That's what they say. And we should trust them on this because...? More importantly, even if they're doing everything client-side now, why should we trust it won't change in a few years?
Also, they aren't doing everything client-side. For this measure to be useful, it has to send out the hashes, or that there were matches detected, to the company, and ultimately the law enforcement.
> it seems like an apartment complex saying you are not allowed to have fires inside your apartment so they installed a smoke detector in your apartment
This situation is not like a fire. A fire in one apartment threatens the entire building (and everyone in it), and it spreads very quickly like started. This is more like installing chemical sensors in every apartment that will automatically call the police when they sniff out illegal drugs. You have nothing to fear if you're not an illegal drug user... unless the sensor has a false positive against your medication. Or the volatiles from your cooking. Or the cleaning agents. Or a jar with a mix of benign substances that someone sent you to screw with you. Or...
Perhaps people would mind less if they could trust the system will work. But anyone with even little bit of exposure to tech industry already knows that these systems don't work, and data collected is routinely abused.
If you use an Apple phone, you have no choice but to trust them. They produce your phone and control everything on that phone. Did you trust them before? If so, why and why would you not trust them now with this feature? If you didn't trust them before and don't trust them after this feature, you probably shouldn't use one of their phones.
> This situation is not like a fire. A fire in one apartment threatens the entire building (and everyone in it), and it spreads very quickly like started. This is more like installing chemical sensor...
I would consider a sensor that detects drugs in apartments a lot more orwellian and I'm pretty sure that something like that would be a lot more controversial than what apple is doing here. They are not adding a feature to catch people that use drugs, just child predators. There is a massive gap between the two and there are zero people that consider allowing child predators to exist is a good idea. I'm also pretty sure that there are zero people that agree that letting apartments burn down is a good idea.
> Perhaps people would mind less if they could trust the system will work. But anyone with even little bit of exposure to tech industry already knows that these systems don't work, and data collected is routinely abused.
Yep, exactly, this changes nothing really. I hate apple products and I don't use any, but their reputation is the least bad and I trust them more than others.
There are horrible monsters in this world, we don't live in a libertarian society, most people don't want to, and a lot of people will be completely fine with sacrificing client side scanning of their pictures to catch those horrible monsters out there. This issue is not black and white/right versus wrong. There are trade offs and it was a judgement call by apple. Everyone has some line where they would be ok with this feature. If it helps avoid the scarring of 1,000 children per year, maybe you don't think its worth it, but there is some line where everyone concedes. I don't really know what is an acceptable amount of horribly scarred children to trade for not having one's pictures scanned on one's phone but if they keep their word and it doesn't turn into an orwellian nightmare, then I would be incredibly grateful for every scumbag they help take off the streets.
100% agree, this is called a "chilling effect", shutting down discussion and giving more power to the status quo.
These are pressures which likely help keep society relatively stable. People's behaviour is guided and influenced by what other people might think of it, and human ancestral environment up until cities(?) was small groups and families and villages where everyone knew everyone else's business. I don't think we should take it as given that changing that is unarguably for the better. Historically there was always the chance that someone would eavesdrop, or see your diary lying around, or open your mail,
> "And this has prolonged effect on society's freedom, democracy and rights of individual because independent thinking happens when true privacy is there"
Citation needed because "Please don't machine-process my photos because I get so anxious I can't think" sounds like a problem you need to deal with, not a problem for everyone else to deal with. So does "The effect will be also huge. Just imagine what they will do next if they get away with this." - always jumping to the worst possible doom scenario is something people see therapists about.
> "The issue here is installation of Spyware Engine."
This is no more than a bank having to make sure you aren't storing stolen diamonds in their bank vaults, and they've now found a way to check only the things people are about to bring into the bank to alert the bank as early as possible.
But you are right - as long as they have the right to push code to your device at any time, it isn't really yours. Shame it costs so much to keep the bank's vaults in our pockets at all times.
They are talking sanctimoniously about privacy out of one end of their mouth, put on a family friendly face by banning anything offending prude sensibilities, while having no qualms about doing as much business as they possibly can in China. Now this.
They are the ultimative dystopian corporation, I find them even worse than Facebook in that regard, at least Facebook has some interest in free speech and does not operate in China.
I guess the reason I am so angry is that I like the devices and engineering marvels they can produce. But I cannot be a customer of this.
That wasn’t really their choice, and they’ve been trying to find a way back into the market since the day they were banned. They still make billions of dollars every year on ads from Chinese buyers that run globally.
But here they'll scan the local files on your own device, that you supposedly "own" (but not really, because Apple already locked it up to make sure you can only install stuff they have approved and taken a cut off)
only the files you upload to apple photos.
Turning off cloud photo syncing will disable the scan on both Android and iOS.
Further, they are looking into doing something similar as Apple:
https://bgr.com/tech/whatsapp-refuses-to-use-facebooks-creep...
I'm looking at doing the same and I noticed Nokia has published the source code for the 8110 (I believe kaios runs ontop of Linux)
Expensive phones triangulate via towers (fast!) and use that to bootstrap the GPS unit. Some GPS units don’t support that bootstrapping and are, thus, slower.
Theoretically it might be possible to combat cell tower triangulation by having the modem only respond to one tower at a time and insert artificial latency to make it harder to work out your distance. But, another convenient fact: cell modems are all closed source, proprietary black boxes. They also have DMA (Direct Memory Access) to your phone.
Perhaps I should be wearing a tinfoil hat but these don't seem like coincidences to me. They seem like intentional measures to weaponise a tool in our everyday lives.
Nope. I have several dev devices. None of which have an iCloud account setup.
Didn’t do anything special; just skipped it at setup.
However there are a couple of new "Fintech" banks in the UK that are mobile app only. No website and I assume very difficult to get anyone on the phone. That seems crazy to me. Access to your bank is at the mercy of Google/Apple.
Deutsche bank uses PhotoTAN, which supports hardware TAN generators [2].
Some banks use different but similar TAN generators (e.g. SecurePlus [3]), and I also witnessed Postbank accounts operated with USB-based TAN generators [4] called "BestSign".
[1] https://www.voelkner.de/products/476429/REINER-SCT-tanJack-o...
[2] https://genostore.de/db/phototan-lesegeraet/92/phototan-lese...
[3] https://cb.kobilshop.com/secureplus-generator/1/secureplus-g... securePlus Generator
[4] https://www.seal-one.com/devices.en#DiVc3100KLink BestSign
On the positive side, a stand-alone air-gapped TAN generator feels much saver than using a (possibly back-doored) smart-phone to do on-line banking.
They have done all kinds of dark patterns, for example they update the website (which eventually becomes a mandatory update), but that requires relinquishing the hardware token, or they update their mobile app, and if you even try to log-in into the new app they automatically cancel your token, etc.
When my token's battery died, they didn't want to give me a new one. They said they don't offer the service anymore. Only after escalating N times and explaining that I need a hardware token because their SMS-based 2FA didn't work internationally they gave me a new token. Suddenly it wasn't discontinued anymore. Now they moved off SMS-based 2FA to some mobile app, so I suspect next time I need a token I won't be able to get it.
Make no mistake, enjoy your TAN generator while you can, because you will not be able to enjoy it for long.
In the country I live in right now, the only way to get a proof for your COVID-19 vaccination is if you "voluntarily" enroll into some phone-based authentication scheme with your government that requires a modern, non-jailbroken iOS or Android device.
You have to log-in to a government website, and you do it either through a mobile phone registered with the government, or with a smartcard and a proprietary Java application. Pick your poison. (And they don't want to issue new smartcards either.)
I've run into a few that require 2FA - either SMS or email, your choice.
[1]https://www.mediamarkt.de/de/product/_reinersct-tanjack%C2%A...
I’ve been looking at trying something like a Pixel but with Calyx as a first step while waiting for the PinePhone and the Libre M to mature further.
This can't happen fast enough... though I can't get behind Purism/LibreM it's just too fucking expensive, basic privacy and ownership of your device shouldn't be a privilege.
I didn’t even realize apple had the ability to do something like this? Can someone explain how this is even possible for them to do? This wasn’t an update was it? Just a flipped switch?
And your final question "What else are they capable of doing without my permission?" --- the answer is anything they want. Concretely, what are the capabilities installed NOW that they can take advantage of, who knows exactly?
As an example, I do remember a looong time ago Google remotely removed apps from Android phones, pretty sure Apple could or might even have done so as well.
Another solution for apple is to simply revoke the codesigning certificate that is used to sign the app, which will render the application un-runnable.
[0]: https://www.macrumors.com/2008/08/06/apples-ability-to-deact...
This means they can intercept even e2e encrypted messages such as signal, if they wanted.
When you control kernel, you can do anything, to anything running on that system.
For comparison in a Linux distro:
- everything is built from source on distro infrastructure, users can inspect any an all source code of everything running on their machine - software updates are transparent and not enforced, user can read changelog or compare source code of the updates - software updates of individual packages don't usually go directly from upstream open source software developers but via a package maintainer in the distro, for each distro - if an upstream project introduced fishy stuff like Apple is doing would almost certainly be noticed by either one of the package maintainers or users due to changes in the source or in software behavior, alerting others to do source code analysis and stop the attempt from affecting users
Nothing technical prevents them from putting something in ring zero which does silent updates without announcing them, but that's not what's happening here.
Basically, is it with interacting with Apple users if you are worried about that problem?
Here it is: https://puri.sm/products/librem-5 and https://www.pine64.org/pinephone/.
It also supports the Librem 5, but also the Oneplus 6 and 6T
* benchmark results faked by software
* personal data collection
* backdoor
All of these do not appear on the English page so there's a link to the French one : https://fr.wikipedia.org/wiki/OnePlus#Controverses
It costs $50, though.
"Sailfish X is currently available in the countries of the European Union, Norway and Switzerland ("Authorized Countries") and the use of our website and services to purchase Sailfish X outside of the Authorized Countries is prohibited."[0]
I think it is because of the USA's allowance of trivial patents which act like a trade block.
In practice only the act of buying the actual Sailfish OS license in https://shop.jolla.com needs to be done from the the EU/supported countries. That can be done via a EU VPN and none of the Jolla provided services (RPM repositories needed for system updates, etc.) are geoblocked. I went to Japan with Sailfish OS phone twice with no issues at all.
For reference:
- getting SFOS from outside of EU:
https://together.jolla.com/question/184861/ask-purchase-sail...
https://forum.sailfishos.org/t/has-jolla-abandoned-sailors-i...
- stats from the community Sailfish OS software repository showing a lot of traffic from countries outside of the officially supported area
I have recently installed it on the Xperia 10 II you mention as well use a couple Xperia X devices in the family. Use the Jolla 1 & Jolla C before back when Jolla still manufactured their own devices.
Frankly, before some of the PinePhone distro mature, this is the only really usable Linux distro based independent mobile OS. And it has been available for years, yet people don't seem to be aware of it or seem to ignore it for some reason.
And if you have any questions about Sailfish OS, fire away! :)
No idea about the ToF sensor - I woukd guess it could be used unless it nerds some weird Android blobs to function.
And while I agree that the closed source parts are stupid I still think Sailfish OS is a good stepping stone to full open distros and hardware once they reach sufficient maturity.
Also have they resolved the kernel updating issue? All phones which ship with Android 11 or lower are unable to have their kernel updated because each device uses a modified kernel[1]. Phones which ship with Android 12 like the Pixel 6 all use the same Android Common Kernel, so it's enabled Google to guarantee the Pixel 6 will have at least 5 years of kernel and OS updates for the first time in Android's history.
https://jolla.zendesk.com/hc/en-us/articles/201440787-What-A...
You can even install microg or full blown Google apps if you really want & are fine with the implications (still less problematic having Google stuff in the emulator than on a native device IMHO).
There have been actually also some attempts to get Anbox running on Sailfish OS as the community ports dont get the Android emulation layer, only officially supported devices.
As for major version kernel update issues - they did not, but I am not really sure it's that problematic in the end.
So basically they base their port on the best kernel version + blob bundle from the Sony open device program at the time, then stick with it. Apparently the way updates of these low level things work on devices originally shipping with Android are too stupid and fragile to make supportable over the air updates possible for Sailfish OS.
They could port the adaptation to newer bundle from the open device program and either mandate re-flashing or support two versions based on different kernels on a single device. Both not very good options imho.
Still, not updating the major kernel version does not mean the Don support the devices or newer rebuild the kernel. All the Xperia devices in the program are still getting OS updates, including security fixes for the kernel. They actually only recently dropped support for their Jolla 1 handset released in 2013.
It's just when you really need the latest kernel features or the most advanced Android emulation layer you might need to get the most recently supported device.
Thats what kept me there for all those years. :)
Did you take PureOS into account ? It is the Linux OS installed on the Librem 5 phone. https://puri.sm/products/librem-5/
And how about specially crafted 'normal' pictures sent to individual to trigger the system? Special mind would have a perfect tool to get some one down / put into the troubles.
Once things like Apple is doing become normalized, the next step becomes making it illegal to distribute or use software that doesn't do it. So running your Linux phone will be illegal, and why would you want to do that anyway, except to look at child pornography?
Similarly, non-sanctionable digital currencies like Bitcoin will be made illegal.
Avoiding modern tech is a workaround, but will be get increasingly hard as the world becomes increasingly reliant on it. (Cash, also, will have to go away.)
I’d like something like my 4 inch iPhone SE but I know that is impossible. How about 4.7 inch range like the new iPhone SE?
Edit: If anyone is ever reading this and looking, I went with a Pixel 2, has a 5 inch screen and a good camera. I'll see how it goes. Will be nice to not be locked into Apple anymore.
https://www.phonearena.com/phones/size/Google-Pixel-4a,Googl...
Keep in mind that bezels have gotten much smaller over time.
There is librem and pinephones and they don't look bad, but don't expect many apps and interoperability. Some people regard that as a plus.
There's the Pinephone these days, but it's not ready for general consumption.
If someone takes a new picture of a child being abused, that won't be on the database so won't get flagged.
Are they expecting child-abusers to take photos of existing photos of children being abused?
Or are they hoping to catch abusers sending each other known photos of child abuse unencrypted? Because that's:
a: stupid (the even-remotely-smart abusers will encrypt their abuse) and
b: rife for mis-use (I, using a non-Apple device, send your Apple device a photo of abuse. You're now flagged as an abuser and your life shuts down)
As others have said, I don't think this has anything to do with child abuse, because it plain doesn't work as a method of preventing child abuse. It totally works for flagging "politically unacceptable" images, though. Maybe the CCP got fed up of all those memes?
I also hope nobody here ever gets mad about the rampant child pornography going around on platforms like Kik, because that would be incredibly hypocritical.
0: https://www.theverge.com/2014/8/5/5970141/how-google-scans-y...
1: https://nakedsecurity.sophos.com/2014/08/10/microsoft-scans-...
Also your links say that Google and Microsoft scan e-mails on the cloud. Not phone storage as in Apple's case.
Another confirmation that Google and MS have access to the data you store in the cloud and Apple does not. It's very unfortunate that scanning this locally is viewed as a negative in stead of a positive.
FBI 'persuaded Apple to halt iCloud encryption' - https://www.bbc.com/news/technology-51207744
(Honestly Apple probably doesn't really have an option -- the govt wants this, and I'm sure they can exert all kinds of leverage to get it if Apple doesn't comply.)
Android does not scan your phone. Google scans content uploaded to ITS machines - because content hosted on ITS machines is a liability.
You can't be in hot water for hosting/distributing CSAM if it never gets on your servers/network in the first place.
There's way more to it than appears at first blush. This is much more than a kiddie porn thing, it's a fundamental shift in the Overton window.
Hash collisions are 100% impossible to avoid. You are mapping an infinite set (the set of all possible images) to a finite set (a fixed length number).
Cryptographic hashes are designed so that collisions are hard to construct at will. But this is not a cryptographic hash at all and I wouldn't be surprised that constructing an image that matches a given hash is easy.
>Hash collisions are 100% impossible to avoid. You are mapping an infinite set (the set of all possible images) to a finite set (a fixed length number).
If you want to be needlessly pedantic I guess. But for 99.999999% of usage hash collisions don't exist in practice.
>Cryptographic hashes are designed so that collisions are hard to construct at will. But this is not a cryptographic hash at all and I wouldn't be surprised that constructing an image that matches a given hash is easy.
You got a cite to back this up? Because they claim otherwise.
Oh right, that very soothing. The very device I spent $1000 on has 0.0001% of ruining my life by causing a no-knock raid due to a false positive. They should put this stuff on their ads, makes me wanna buy more Apple products.
I'd much rather sell all my current Apple devices and permanently switch to linux than do this.
If you're correct, and the chance of a hash collision is 0.0001% then for each hash in the database, that's 200,000 collisions.
Assuming the database has 1,000 hashes [2] and there's no overlap in collisions for any given person, that 200 million peoples' lives ruined.
[0] rough guess based on: https://www.statista.com/statistics/276306/global-apple-ipho... the exact number might be off but I think the order of magnitude is about right.
[1] My gf has taken at least 3 photos every day for the last 5 years at least (so well over 5000 unique photos) - 200 is very conservative.
[2] Again, very conservative. This is a collection of all known child porn images. I wouldn't be surprised if the number is actually two orders of magnitude higher.
[edit] replied to the wrong comment. Bugger. Hopefully contributed to the conversation anyway.
>less than a one in one trillion chance per year of incorrectly flagging a given account
If we gave every single human being an iPhone we'd expect an incorrect flag every 160 years or so.
And I guess we'll find out.
The horrible thing is, of course, that this whole process will probably be automated and there'll be no recourse to a human with any power to work things out properly. If there are thousands of cases, they'll have to take to Twitter to shine some light on it. Except that in this case they're self-identlfying as suspected child abusers. How many people are going to risk the negatives that go with that? Will we ever find out how many people were actually false positives?
Hah. That's an interesting point.
If anything you are encouraging new child porn.
* Microsoft scans and deletes stuff on your drive it believes is harmful [1]. There was also some indication a while back that they would delete 'pirated content' - i.e. anything some algorithm detects you "shouldn't have" [2].
* Google Drive have been deleting 'pirated content' for quite a while based on hashes [3]. I imagine other cloud services do this as well [4] and I recommend you do not sync your important files with such a service.
This was always on the cards. Next it will be 'terrorist materials', then it will be 'harmful content' and soon simply 'DMCA content' stored locally will be reported to the police. Freedoms definitely erode if not defended.
On a related note, projects like Pine64's Pinephone are early stages (read: difficult to daily drive) but definitely in the right direction [5]. Some day in the near future you will be able to get a decent mobile experience without having to worry about spyware/adware/malware.
[1] https://news.ycombinator.com/item?id=27914752
[2] https://www.indiatoday.in/technology/news/story/windows-10-w...
[3] https://torrentfreak.com/google-drive-uses-hash-matching-det...
[4] https://www.extremetech.com/computing/179495-how-dropbox-kno...
You should read the linked article more carefully. It makes an incorrect conclusion. IANAL, but if you read the EULA lines carefully, you'll see thst they're saying, basically, "We might auto-update software, which may incidentally break your pirated version, and we're sorry and don't want to get sued if this happens to you".
I do remember very clearly that Microsoft were deleting pirated version of Office on purpose and this EULA basically said "oops, we told you it might happen".
> This week we received a tip from a reader who was unable to share a link to a screener copy of a Hollywood blockbuster. Instead of a public link, Google drive warned that sharing the file in question could violate its terms of service.
What you linked to says that GDrive prevents users from sharing pirated content, not using GDrive for storing them (or that GDrive was deleting them).
I performed an experiment to test whether GDrive delete pirated content - they do.
Pirate a copy of some modern Disney/Pixar film and upload it to your GDrive. Within a few hours it'll likely get DMCA'd.
EDIT: There is no requirement to share a link, Google will scan your drive once it processes the video.
There are other such reported cases of DMCA violations, etc, so this isn't isolated [1] [2].
[1] https://support.google.com/accounts/thread/21194028/drive-sh...
[2] https://forum.ragezone.com/f872/thats-kinda-messed-rh-bin-10...
[1]: the guy was sharing Windows ISOs.
[2]: Reqvim seems to have been sharing a modified binary (the original was created by someone else).
If you've experienced it I'm wondering if you might have uploaded it to a shared folder. But I'll keep your anecdote in mind, not too interested in reproducing it on my primary account, and not sure if enforcement would be different if I were to use a burner.
I'll also note that as mentioned by a commenter in [2], Google isn't required by DMCA to remove content on GDrive that isn't being shared, though this might be a user's breach of TOS.
The tools already exist and the legal framework is essentially in place - every time you go through border security you essentially are told "we can search everything on your person (including your device) and there is nothing you can do about it".
It's not such a stretch that some 'national security' enforcement is put in place. Imagine having the same applied to your work place or public transport. Essentially the same that is happening with vaccine passports, where the number of places you can go is slowly eroded to the point that you can't viably go to any public place any more.
New Zealand for example for many years have had the ability to perform a 'digital strip search' where they can force you to unlock your device [0]. If you live within the Five Eyes you can almost guarantee this is already in effect for you or will be soon.
On the plus side, this has led to some 'hilarious' stories where the TSA border security were physically searching a person for Bitcoin [1].
Fun times to live in.
[0] https://www.washingtonpost.com/news/morning-mix/wp/2018/10/0...
[1] https://www.forbes.com/sites/kashmirhill/2014/03/03/why-the-...
I know the feeling.
And while we all agree about the fact the child porn should be stopped and people should go to jail for it, apparently "what happens on your iPhone stays on your iPhone" only until Apple says so.
Do you suppose any government is going to pass laws which prohibits companies from scanning photos for CSAM either on-device or in the cloud? When half the time it's being done at the insistence of those self same governments?
I was lucky to have not one but two uncommon surnames. (grandmother remarried, children got an extra surname). There are only a dozen people with that combination of surnames.
How can you be sure of that?
French immigrant mother so French first name and middle name, and then an uncommon British (Welsh I think, dad's parents didn't learn English until they got to secondary school) second name.
For the benefit of non-Polish-speakers:
- Nazwiska - surnames
- żeńskie - female
- męskie - male
He was denied an operation at the bank once, but was quickly able to clarify the issue and unblock it (probably he got lucky).
But he had to have himself removed from the white pages and the ring label (ring label? whatever, il citofono :D) because of violent threats.
This is a major part of the problem with algorithms: what if the issue is "computer says no" and that's the most detailed explanation you can get?
In the end, they thanked me for cooperating (they were actually quite friendly and polite in general, not like going I to the US, for example) and explained that they were looking for someone with the same name as me.
After a bit of investigating I found out that I had been placed on the Credit Industry Fraud Avoidance System because different spelling of my name had been used in previous applications. This was seen as an attempt to get past credit checking, in reality the people entering my name in to their systems mis-spelled/mis-typed my name. As a result of someones carelessness I was flagged as being risky.
Casual identifiers have to be stored as plaintext. To facilitate record matching (linking) across data stores.
The only way to enable full field level encryption for data at rest is to issue everyone GUIDs.
If we're not willing to do that, for whatever reason, the status quo (a boring dystopia) will persist.
cite: Translucent Databases
Let's see if fear of AI is greater than brand loyalty and convenience. An unstoppable force meets an immovable object.
Apparently Apple has a team working as AI police. These people decide who gets reported.
> While noting the 1-in-1 trillion probability of a false positive, Apple said it "manually reviews all reports made to NCMEC to ensure reporting accuracy." (https://arstechnica.com/tech-policy/2021/08/apple-explains-h...)
Don't even get me started with the 1 in a trillion probability, we know adversarial images could be created that will trick the system even when you don't have access to the model.
I can't precisely buy a new phone in a whim, but I'm not trusting my well-being to any "one in a trillion" in the marketing of a company.
Not arguing with the general sentiment though.
I don't think so - I think corporations and governments are getting better and better at PR and so, we'll see, slowly but surely, more and more features like this.
It could maybe be used to perform a kind of DDoS attack on the human verification stage by increasing false-positives, but I doubt a non-child-porn image would fool a human into thinking it looks like child porn just because of some carefully-applied noise.
Also, in what conceivable way is this like Minority Report, ie three psychic humans, floating in a pond, hallucinating the future?
Wow, come to think of it, will the hash database on your device change when traveling between countries? A popular photo of two women kissing on your phone could be perfectly legal in the US, but suddenly get you jailed if you travel through Saudi Arabia and forget to delete it.
I sense irony. But I'll still reply:
> Hash database checks would be a privacy friendly alternative to custom agents physically accessing your phone when you enter a country
The border agent could always claim that their system detected something and therefore they need to look through all your private files.
Is there one? Reading a BBC News story on this, it sounds like checking is done when you try to upload an image to iCloud, so it's on the server.
THIS THIS THIS THIS!!
How will this work when traveling? If I have legal non-CSAM pornography, but I travel to another country where pornography is completely illegal, will I get arrested?
Who controls the hashes? When do the hashes get updated?
Fuck I feel so betrayed by Apple.
But virtually zero mentions of the U.S. Congress, President, judges, political parties, etc. The folks who are presumably behind this new Apple feature. (Does anyone see a "Add Evil Feature...Profit" story here, that does not require massive government arm-twisting?)
IANAH (...Not A Historian), but I'll guess that if the well-educated citizens of a country do not believe that it is a real democracy - or feel that being politically active on issues that they care about is an unacceptable burden - then (at best) it very soon won't be.
1. It's been turned on for all major cloud providers for some time. Upload something into google cloud/photos/dropbox? Yeah, the system's there as well. It just seems most people have been unaware of this so far.
2. It's only for the US. From my understanding Europe and ROW doesn't get this. At least for now.
3. I don't like this way of doing things, but I do understand that there's been major push from the US Government for this kind of tech. Basically, the tech companies had two choices: either add an actual backdoor, or provide another way of checking the content of all their users. They went with the latter.
4. If anyone has a problem with this solution, they should contact their US Representative, as they've been pushing this like crazy "for the children" :)
5. it's not scanning your local files. only the ones uploaded to apple photos. just like all the other actors.
AFAIK the EU is actually pondering legislation to make proactive scanning for CSAM mandatory. Right now, in the US at least, proactive scanning like practically all cloud providers do is not required by law.
i fear more governments will take this position now that the big ones managed to push it thru.
The issue is installation of Spyware Engine, not it's current usage.
The issue is your calling it "Spyware Engine" as if it's anything that wasn't happening before - mesh networking for "Find my phone", watching bluetooth and wifi and GPS locations and reporting back to HQ for things like mapping and traffic monitoring and route planning, and possibly advertising, these are much more "spyware" than this is - they happen more routinely, to more people, and report more back.
UK ISPs have been legally obliged to block certain websites from access for years, Google knows where you live from nearby WiFi SSIDs and can correlate all your browsing traffic back almost entirely to your real identity all the time, GMail and Office365 and FaceBook and Microsoft know everyone you contact and what you talk about and therefore what kind of relation you are to each other, all the big providers know where you live and where you work and whose house you spend time in from location tracking and correlating cookies with IPs. This thing from Apple is way more narrowly constrained and scoped than any of that.
Two things. First, this feels like the outrage about the CPU ID in Pentium III's, until someone figured out it had been there all along in Pentium II's.
Second, are there any statistics which show that this scanning has been effective? Searching reveals a lot of conflicting information about child predation statistics.
That was the wrong decision. Had they just removed the strong encryption of cloud content, this would be the standard case you are referring to.
Now, since it's local, the question is no longer what it's currently being used for, but how long before they are obligated by the government to activate it for local files also.
All in all - it’s very possible.
Suppose your target post a picture from their iPhone to a social network. You just have to add this picture perceptual hash to the offensive database for bad things to happen to them.
To do this : you grab their picture, overlay some offensive content on top rendered very transparently so that the new image is like 95% original + 5% offensive content. You also generate a second image with ratio 98% original + 2% offensive content.
Then you post the pair of images on a public site that usually get scanned for offensive content. (You may even add the original image for good measure). Given that the offensive and original image can be exactly reconstructed, and that even at low transparency the human eye can still distinguish the offensive image it won't take long before all images get labelled as offensive. Their hashes added to the database, and a collision with the picture of your victim registered.
Launch a script and do it for every pic they post. Of course there are plenty of variations of the above technique.
Have any other big tech companies that scan for illegal images been shown to be vulnerable to this?
But if you can't store the data and you are only allowed to store hashes, you become dependent of the quality of the hash database which is troublesome to verify, and therefore susceptible to poisoning.
How hard the game will be for the various players is not really the point, given that it's a game that shouldn't be played : only Apple know the rules and can verify them, and not even Apple should have legal access to offensive raw database to check that it hasn't be poisoned.
If they use some neural networks, it can easily turn into a PR disaster, once someone take their weights and use a Generative neural network to hallucinate offensive content based on the weight they release.
How long until this is reality?
[1] Collateral Murder is the war crime the US committed and was leaked via Wikileaks and is the reason Assange is still behind bars.
Are there nonpublic images from Jeffrey Epstein's blackmail collections already in the database? Is there any system that could be developed to trustworthily prevent this situation without distributing child porn?
If anyone thinks or says otherwise they ought to think about what else they would agree to where the other person can do what they want to them without their consent.
If this was a problem, would we not be seeing a slew of complaints about innocents being dragged through the mud with OneDrive and PhotoDNA? The only thing unique about Apples implemention is that it's client-side.
In the story it didn't became a well known issue since it would happen only to few unlucky individuals.
The same goes for the unintended asset freezing.
- You don't like storage provider - you don't use it
- You don't like ecosystem/smartphone provider - you don't use it
On top of that you can opt out even now by disabling updates it just means that you won't have access to the newest iOS and take risk that at some point software developers will stop supporting OS you decided to stick to.
This just doesn't work like this in our day and age. We are one ecosystem (android) away from complete domination of this scanning technology. You could argue that I could use a librem or something but at that point all librem users automatically become suspicious because "all major manufacturers have this, he probably has something to hide".
you could as well live in woods away from society, but that's obviously not the solution to bad laws
If Google decides to implement this, which doesn't seem very unlikely because of the optics of NOT implementing this, people who aren't tech savvy won't really have a realistic way to opt out.
I just don't think there is very much evidence that in practice false positives are a big issue and the article is very much pushing that argument.
That said, many hosted providers/social networks have similar features - they just have server-side implementations and might not have felt the need for disclosure.
I would agree on a purely technical level your phone is already pwned by Apple so worrying about this on a technical level is closing the barn door after the horse got out. However from a social perspective one of the holdouts against photo scanning has stopped being a holdout, and doing things from client-side makes it seem less wrong to do other things client-side.
Besides the privacy stuff, this also is a bit more of a slide towards software that you purchase being ultimately controlled by and for the benefit of parties other than yourself.
In practice any major cloud provider is going to or is already doing this. We need a better regulatory approach, it isn’t practical to put the responsibility on providers.
https://www.macrumors.com/2021/08/05/apple-csam-detection-di...
"CSAM image scanning is not an optional feature and it happens automatically, but Apple has confirmed to MacRumors that it cannot detect known CSAM images if the iCloud Photos feature is turned off."
We won't get one though, because the "think of the children!!!" crap is extremely pervasive and anyone going against it will be smeared as a pedo defender.
In that mindset, the answer is absolutely to combat and to attempt to change broken laws first.
Also, it isn't just "think of the children". For instance, there have been some _terrible_ proposals under the banner of Right to Repair. People tend to not want to invest the time in understanding the ramifications of the actual proposals, and instead vote for or against the concept. One of the reasons ballot measures are both empowering and terrifying.
Good regulations take time and care - and generally, less is more.
That has to be done on the phone because Messages is end-to-end encrypted. If they are going to have to have hash matching on the phone anyway for that, it makes sense to also use that for checking images that are to be sent to the cloud.
Presumably it is client side so that they can do anonymization/encryption of photos on the server, and treat any data access outside the account and account they have shared the photo with as an audited and cross-organizational event.
But if you want to use another hosted service, you can... and likely get their implementation of a similar system. Presumably this is US regulatory compliance.
This is going to bite more innocent people through false positives than criminals who already know how to get away with these things.
It is not a ML model but a list of known image hashes, and is only enabled for US-based accounts, furthering my suspicions this was minimum-effort for regulatory compliance.
Note they _do_ have a feature (also announced today) that uses ML models, but it is meant for local filtering and parental controls/notifications. This feature is also US-only and the parental notifications policy is fixed and age-based. I believe this is both to fit into regulations (e.g. US recognition of rights based on age) and into cultural norms.
I suspect they will have different rules in different jurisdictions when this rolls out further in the future.
[1]: With separate key escrow HSMs for account recovery and legal compliance with e.g. court-ordered access.
the tech runs locally, but only on those photos.
and in real life governments elected by the people have been pushing for this for years. the result has been google and all the other cloud providers already implementing this. apple was the last big one to hold out.
will they expand this in the future? sure, whatever. the system is so broken, and i’m so powerless, that at this point in time it doesn’t matter what i want.
at least it will only apply to the US. the ROW is spared. at least for now.
Even if the hashing and matching happen on the local device, a match can only be revealed server-side. The hash database distributed to local devices will be blind hashed with a server-side secret key and the locally derived hash match will need to be decrypted with that key to be read by Apple. So theoretically if the local device doesn't upload content to iCloud, no content matching can be revealed, even if the hashing and matching has been done locally.
Of course, you also need to trust that Apple won't be uploading those locally derived hashes to iCloud without the user's permission if iCloud backups are disabled.
[1]: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
You are consenting to the upload and are aware that it can be searched so not a legal problem.
Law enforcement will also review the picture in the cloud instead of busting your door to search your phone and the whole scenario from article.
When they do it client side they can't just upload your content on hash collision. (or maybe they do which is also a problem in itself)
- The scanning will be performed only if photos are to be uploaded to iCloud.
- The database will be encrypted multiple times in a way that it can't be clearly read.
- There's not notification to Apple in any way in case of matches.
- Instead, each match result is again encrypted in a way that is inaccessible to Apple and uploaded together with photo
- If there a lot of positive matches, they eventually will become able to decrypt it. That's when they will do manual check, lock the account if it is correct and notify authorities.
This is just another case of, 'your phone' is not yours. People stores their lives and thoughts in a device that they have no control of.
I am not against those use per se. I am worried about what it means legally. The minimum that we need is a more clear legal framework. This is a private company accessing personal data without consent and without any formal requirement from law enforcement.
This is specific to photo synchronization to the cloud, which is an optional service.
In that sense, it isn't really doing anything different than say a OneDrive photo sync app - except it is doing the check on the client side, rather than having the server decrypt to do the check.
It is part of the default install and setup wizard, but thats a battle we lost decades ago in the PC world. You still have to opt into iCloud synchronization for this to be enabled.
Yeah, the only thing unique is installation of Spyware Engine in personal device that will do 'who knows what' later. What can possibly go wrong?
Other hosting providers do similar scans, but they do not do them within the client-side component. Presumably, this is done to give the hosting environment no access to the actual data (e.g. data access such as by subpoena becomes a cross-organizational auditable event)
...an anonymous member of a child pornography forum used the handle "$NAME_$YEAR" and had his location set to $TOWN. The innocent bystander's middle name is $NAME and he was born in $TOWN in $YEAR but had not lived there for decades. He only learned about this absolutely nonsensical justification and got his stuff back by hiring a lawyer who requested access to the DA's investigation file.
If you can get a judge to sign a warrant with flimsy non-evidence like this, what will they do when the robot says "95% match detected".
This gives me pause...
I can only think of two reasons for this:
- for a future expansion into files that are not uploaded, even with cloud services turned off
- Apple doesn't want to foot the energy costs of doing it
Finally, it should be noted that this will also apply to macOS Monterey, not just iOS/iPadOS.
If they enabled E2EE iCloud after this it'd be just for marketing purposes, because client-side encryption is being circumvented it would render the whole promise of privacy as a lie.
Client side encryption of what? A plain photo? That you designate to upload to cloud? At which point it's encrypted, or hash-checked then encrypted. Where's the circumvention?
Their reasoning seems to be that it would break E2E encryption.
It's about getting labeled, for whatever reason as "high risk." I flagged some paypal thing years ago, when I moved countries. It was too much trouble to sort out, so I abandoned my $20 and moved on. Paypal has me in a no recourse bucket. Most transactions run through such risk assessments. False positives are tolerated. Child pronography scanning, as this post lays out, will have similar issues. We're already used to credit scores and insurance behaving like this.
I'm not diminishing the privacy aspects, but this neokafka world of automated, risk accessing bureaucracy is also a world we're constantly walking towards. It's also bad.
I'm worried this will apply to more seriously some day. "This algorithm says you're guilty of a crime, and it's 99.8% accurate, which we've determined is acceptable, so there will be no way to appeal your verdict."
>Apple says that it will manually review each report to confirm there is a match. It can then take steps to disable a user's account and report to law enforcement.
They say all reports will be manually reviewed, so the blog post, while painting a scary picture, doesn't really paint a picture that matches the reality of how this works - ie automated systems flagging up with no human intervention. I know the blogger tries to illustrate how human intervention isn't enough, but in this case we're talking about a human just comparing an image with another image, not a hypothetical lazy legal department "playing it safe" instead of doing due diligence.
This technology benefits practically nobody, consumes resources, and is bad for the reputation of the "privacy" aware company.
If something like this is to be implemented the door is opened for scanning everything else.
Here is a question: If I know my photos will be scanned for child pornography, would I keep child pornography in my phone?
I haven't seen anyone who likes this direction taken by Apple. Why would they do something that has the potential to detract from their product demand. Apple always said "We're different - we sell you our products, we don't sell your data." Now, they're giving away the data and for what? Child safety - I don't buy it.
I remember when Apple refused to help law enforcement unlock the iPhone of a mass shooter is California. Why the change of tact now?
They don't want child pornography stored on their servers?
"Apple’s system is less invasive in that the screening is done on the phone, and “only if there is a match is notification sent back to those searching”, said Alan Woodward, a computer security professor at the University of Surrey. “This decentralised approach is about the best approach you could adopt if you do go down this route.”
So it will only happen when you upload data to the iCloud. So the question is then: is it okay for apple to make sure no illegal content is stored on its servers? I think you could argue both ways but to me at least it is less privacy invasive than it seems.
it is about expectations of privacy, in the same way you can expect less privacy in a public space in the real world. It should be clear by now that you can expect less privacy on the public web. Of course a counter argument is that is is perhaps more like renting a hotel room than it is a total public space. But you can also expect less privacy in a hotel room than you can expect in your own home.
For many years privacy and freedom of speech has been a hallmark feature of the internet. I was (kinda still am) a big proponent of this. But we have seen what will happen if you take it to its extreem. From disinformation campaigns to illegal activities. So to me (and incidentally Facebook) we should get better (inter)national rules about how we want to use the public space of the internet. Just in the same way we have rules and laws in public space.
Internet is blending more and more in to the real world. So governments should make it clear in what way we can expect privacy and free speech and where we can expect it (self hosting, public platforms etc) and what responsibilities companies have to enforce it.
The hardline arguments of privacy advocates are the same that the NRA is using to prevent any form of gun control. It starts with A what is them going to prevent to do B. Well rules, regulations and laws. You should be very aware that privacy isn't watered down and law enforcement should make sure they don't mis use what rules there are. But to me checking hashtags for know child abuse photos is the same as doing a basic background check before giving someone an assault weapon
That being said it will stop no one from encrypting images before uploading it to iCloud.
Now that Apple has some form of client side verification, an authoritarian government with enough economical clout can preassure apple into extending the system to look for things beyond child pornography. The govt could even just provide apple with their own hashes which may or may not be CP and mandate that Apple verify these hashes too. So in the end it all depends on how well apple is ready to push back against such government and if they are willing to take a hit to their profit margins by being effectively banned from a country
And suddenly I remember that Apple is playing by Chinese rules in china any way.
https://www.nytimes.com/2021/05/17/technology/apple-china-ce...
The issue here is what they are going to do next without telling you.
The thing is that they do their best to make sure that they do as little as possible of server side processing. Tasks like sorting photos, face recognition is all done locally as to ensure misusage and ending up with a big database of all sorts of data.
They have chosen to do the same route with a very specific set of content checking. of course this tech can be used in malicious ways (banning books and what not) but if you live in a country that has a government like that, best is not to trust any device that comes with software preinstalled anyway
Well "It checks ... " is exactly what spyware does, at least this part is Spyware Engine. Which part of it is not spyware engine? Did you see the source code?
>The thing is that they do their best Yeah, right. Or so they say...
>of course this tech can be used in malicious ways
It is already used in malicious ways, it works as Spyware Engine and we have no information about what else it does under bs umbrella like "children protection" or any other bs they would invent to justify Spyware Engine legalization attempts.
However, the public doesn't (and shouldn't) have the ability to inspect these images to verify that non-CSAM politically threatening images are not included. (ex. pictures of politicians doing cocaine with adult prostitutes, or images of war crimes committed by US troops)
I think that distinction is important. Because even if the NCMEC database is only child porn right now (which we cannot know), the public will also not know if the scope of the database later changes because now Apple will call the police on whoever possesses an image that matches this database. Adding this capability increases the incentive for misuse of the database.
A better (and more related to his issues) scenario would be Apple automatically kicking you off their platform and you losing access to all your data, including your passwords which you stored in iCloud Keychain. Because that is an actual realistic scenario.
https://en.wikipedia.org/wiki/No-knock_warrant#Controversy
Key quote near the bottom:
Due to errors or acting on bad or faulty tips without double-checking information, Chicago Police Department has raided many wrong addresses.
So yes, raids without double-checking the information (i.e. looking at the picture that lead to the tip) has been happening quite recently.
Sometimes, this is the only substantial allegation.
Show me the man, I will show you the crime ( Stalin? Beria? )
If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him.
Cardinal Richelieu
I somehow doubt it.
In practice ?
(Edit: This is for the iCloud scanning. The (i)Message warning around sexting is a whole other issue)
That shouldn't make anyone sleep soundly at night. The person at the other end has no incentive to not turn your life upside down.
"Rest assured our system is safe, a random guy working in a digital sweatshop at the end of the world will get to see your naked baby pictures just to make sure you're not a criminal"
One would hope that such checks should be minimal and therefore performed by suitably qualified (and equally audited) personnel. As you allude, it is more likely that such work is outsourced in the same way that the manufacturing of the phone is.
[1] https://www.theguardian.com/technology/2019/apr/11/amazon-st...
What will the job interview sound like for this job? “Do you have any experience in recognising child porn?”, “yes, many years, and great enthusiasm for the job”.
I’m looking for my old Huawei phone now.
So Apple does have access to everything you store on your phone? Either they're lying about being concerned about privacy or you don't have your facts straight.
But that’s a stupid way around it, surely. Child porn is surely, and tragically, in continuous production. And really, it’s the production that is the biggest issue, this is the continued abuse of children. And that’s the one bit the algo won’t match against?
When I first read this I thought it was impossible. But it seems like, while Apple’s total revenue from the US is double that of China, actual iPhone sales volumes may not be as far apart. The numbers aren’t publicly available, but Bloomberg has an estimate as of 2017 showing US iPhone sales of 70 million vs 50M in China:
https://www.businessinsider.com/apple-iphone-sales-region-ch...
It looks like iPhone sales were going down in China as of then, but their GDP has also grown rapidly in recent years and I’ve seen other charts showing Apple’s China revenue basically increasing in concert with Chinese GDP, so it may not be a stretch to say that China is the biggest iPhone market today. Whoa.
[0] https://www.youtube.com/watch?v=dbYZVNSOVy4
Edit: tracking -> scanning
Why? do car companies feel also compelled to give your car's location to all the governments? There is no need for apple to do anything, it's a choice, they are doing tit-for-tat with governments. That's always nefarious
And no matter what move I make next, it’s going to be a downgrade in hardware specs. What a bummer.
And here’s another thing. In the OP, they make it sound as if the imaginary mark was a female. We all know that females won’t be affected in any way. Most females will probably not even care about this. Because they don’t know what it’s like to be nervous around children or to have to make sure you are never alone with children even by accident or happenstance because everyone assumes that you are a pedophile, rapist kidnapper. Leery looks from adults all the time whenever there are young girls around. It’s a uniquely male experience. It’s also uniquely male to have your guts cut open and ripped out by your cell mate. When people start being brought in for a hash match and then they find incedental images of young girls that look like they might not be 18, innocent MEN are going to die.
You create anonymous account, especially pretending to be underage, you send blacklisted pictures to the target. It will be automatically stored on the device of the victim at reception and maybe automatically uploaded to icloud for backup.
Now you just have to wait for an automatic apple shit to trigger and have the company and cops ruin the life of your victim automatically.
It is SaaS: Swat as a service :-)
On my iPhone all images I look at on WhatsApp get automatically added to my photos, which in turn automatically get uploaded to the cloud.
But all the things that could conceivably happen could always conceivably happen. My sense is that Apple is trying to placate some groups to avoid providing more intrusive changes, such as being able to decode iMessages and removing end-to-end encryption. While there is a lot of concern about a slippery slope, one could imagine this is actually a path that avoids much steeper, harder to avoid slopes.
Apple’s announcement of a limited program of photo matching does not increase or decrease their ability to scan files in the future, but it’s not what they are describing now, and it’s unclear why they would provide such capability since their brand has emphasized privacy so strongly.
It would really suck if everyone starts implementing this and there's such a simple way to circumvent it.
These are completely different situations. Not to mention how infinitely unlikely it is that _several_ of your photos would be similar enough to known abusive material to be flagged.
>There aren’t enough people (nor PR capital) to look through all of your photos.
This is a nonsense sentence. They ARE going through all your photos, both in this effort, comparing hashes, and for classifying your photos with ML, a highly publicised feature. Both happen on device.
In my country, there was not so long ago a case of a person wrongly accused of murder[0]. He spent 18 years in prison, despite there being no convincing evidence of him committing the crime. The people invested in the case wanted to have a scapegoat ASAP. Cases related to pedophilia are often very emotionally-charged, given the nature of the crimes. It can easily lead to hastily made decisions that can ruin many lives.
> Not to mention how infinitely unlikely it is that _several_ of your photos would be similar enough to known abusive material to be flagged.
Given enough time and enough attempts, there still is a possibility of something like that occurring. And this is with an assumption that the algorithm used will be correctly implemented with no false positives at all – something that I doubt even a manual analysis by humans would be able to achieve given the volume of data to check.
[0]: https://pl.wikipedia.org/wiki/Sprawa_Tomasza_Komendy (in Polish)
And I'm not sure what your main point is. Yes, sometimes people are wrongly accused, even convicted. Is that argument against having a legal system, or what are you trying to say?
I am also not saying that such people should be left free and unpunished, but that the method proposed could cause far more bad than good, in my humble opinion of course.
A legal system of some kind is obviously necessary, but using argument of thinking about `insert a group of people you should care about` can easily lead to abuse of the established laws.
If you're lucky. If you're unlucky, they'll insist that you have it stored "encrypted" in a file whose extension their automated scanners don't recognize and keep you locked up until you produce the password. And insisting that there is no password and that the file isn't an encrypted file (it's some system file and you don't even know what it's for) is exactly what a criminal would say.
Jesus. Lopez. Mohammed. Mark Taylor. Sally Jones. Park. Wong. Kim.
Chuck Taylor used to be on there, but isn't anymore. (not the shoes)
Since there are a lot of alias matches, you get a lot of people who wind up matching when there's no reasonable way that there's an actual match. (i.e., they're happily living in the US and have an ordinary bank account there, not the dead dictator of Liberia, or someone in Columbia, or North Korea.)
I want my home server to work as my personal cloud: music, photos, movies, files collections. And everything should be end-to-end encrypted so that no one has any way to look inside it. With 5G and broadband this might even work without much optimization. Someone knows about such solution?
Both quite concerning options TBH.
What algo is Apple using to compare 'similar' hashes from images, does anyone know?
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
I wonder what would be the defense to apple if they decided to scan the images on your desktop / laptop as well. Is there a limit to how intrusive tech companies can get?
“These features are coming later this year in updates to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey.”
And, consequently, a sanctioned person could probably do the same to avoid sanctions.
Regarding the functionality we lose when using a custom ROM, I think the trick is to use a device from a "modding friendly" brand. Samsung, Sony or Huawei are not modding friendly. Get a OnePlus, Asus, Xiaomi/Poco/Redmi or Google Pixel and most if not all hardware will work fine.
(My experience is based using the OnePlus 3, OnePlus 8 Pro and Redmi Note 9S with LineageOS and a Asus Zenfone 6 with OmniROM. Fingerprint works fine and even more exotic features like the IR blaster on the 9S or the flip camera on the Zenfone 6 works.)
Now, by replacing the original OS you also remove some things that makes the phone better. Eg: the stock camera app, which often gives you better processing (there are modded GCam apps that might help fix this, alternatively you can save DNG/raw files). One useful feature that I had on the Zenfone 6 that custom ROMs can't give me is the ability to stop charging at a certain % (can be done with Magisk, but it's not supported by the ROM itself).
The main problem for me are the apps. Many require Google services and Google's safetynet and both are a problem if you decide to use "pure" Android. Even if one decides to install Google Apps to have the Play Store and their other services, there's still the safetynet issue that gets triggered just for unlocking your device.
This stuff happens all the time. Banks will even shutdown your account for suspicious activity and refuse to tell you what activity was suspicious.
I bet its probably more like Abdul Mohammad and the 100 variations of spelling that in English
edit: as anyone that gets OFAC list updates daily would notice
Also, it’s hard not to read your comment and wonder if you hate a large proportion of the human species,
> I bet its probably more like Abdul Mohammad and the 100 variations of spelling that in English
it wasn't a xenophobic or islamophobic comment nor was it condoning the reality or your experience with it, it was an observation from how you wrote your post and what I know about the world
The law of large numbers.
George Orwell was right, he was just not enough pessimistic about how bad things would get.
The author in question got cut off from transferring funds due to whatever reason (a confusion in identity at this point we understand).
Funds could have easily be transferred to him via crypto.
I used to work in a company doing fraud prevention. The first thing that shocked me when I started in the role, was the sheer indifference people working there had for rejected transactions. The way these were supposed to be handled was like this:
1. Transaction is rejected because of a rule/condition/bank decline 2. Investigate transaction manually and identify root cause 3. Contact customer and provide information on how they can resolve this (ask for more details if you are unsure, mark them as safe to allow transactions as it was a clear false positive or ask them to speak to their bank or payment provider as the system wasn't the one that rejected the transaction)
The general view was though that if the customer think it's important, then they should get in touch with the company and there shouldn't be proactive work done on these transactions since chances are "most of them are dodgy". The problem was that there were thousands of these every single day and a very small team working in that department - less than 10 people. I voiced the fact that even if someone is fast at identifying these and going through them (once you developed experience in identifying the cause of the decline and doing the administrative work of allowing the user to complete transactions, emailing them and then keeping an eye on their account for when they try to make a purchase) they couldn't work through more than maybe 2-300 transactions in a day. That was 2-300 of someone working ONLY on this task, almost automatically and having a deep understanding of what happened in each of them. Realistically it was more around 50-100 transactions per day. The rejection list could have up to 3-4000 entries per day.
I argued for improvement of the rules that were pushing rejects into that queue, I argued for better analysis to be completed before a rule would be added, I argued for accepting a higher risk but allowing lower value transactions through to reduce friction and make the queue more manageable. I ended up getting some of the things I was asking for, but what happened was that the rejections went down to 1-2000 per day and people just assumed now that those are fraudulent for sure since now we're "better at spotting criminals"...
The introduction of another system that employed machine learning made fraud agents even more indifferent towards those queues since now "you can't fool AI". It was a very sad state of affairs and since I left there I seriously doubt that this has improved in any way. I remember the email chains you would get from some users who tried to make important purchases or even just regular purchases and not being able to get in touch with a human. I remember the frustration, the friction and the blanket statements that would emanate in team meetings saying that "it's fine as long as we protect the company from losses, so what if a few people have issues"... Those few people could have been them in other scenarios.
I think this will only keep getting worse and worse with time. Each new software that attempts to bring "efficiency" to some of these types of tasks can cause a huge array of problems downstream. You end up with companies that will have 5 people working in a department that should have 500 because machine learning will "take care of the issue". I know I haven't even touched the privacy concerns that this raises, and there are PLENTY and covered in great detail in this post and in the one made by the EFF, but we're forgetting here that those affected by these things are other people.
And as long as governments will say "You need to regulate this SOMEHOW" and companies will come back with "We'll use the power of AI and ML to train NN to easily identify any problems and thus bring efficiency and safety to all of our customers". We're people, we're not machines and we have a vast array of personal circumstances and elements that make us unique and saying that a blanket solution will "fix all problems" is absolutely inhuman and shows a crass misunderstanding of how these tools work and what their unintended consequences are.
TL;DR - using ML/AI/NN will cause a huge array of problems. We need more people working these queues which increase exponentially so that innocent users do not get caught in your "high-tech solution". For each criminal fraud prevention rules catch, a few hundred innocent people cannot complete their activities or are labelled as criminals until proven innocent.
What you highlighted is the reason why I wrote it. Most of the discussions are of an abstract harm against an immediate one. They other those involved.
My experience isn’t unique. It’s fairly mild.
There are people who have gone to jail because of a mistake by an algorithm. And many who have been arrested. Here’s one with Apple involved, https://www.businessinsider.com/tech/a-teen-is-suing-apple-f...
The harm is real. And I have been on both sides. I am a survivor of childhood sexual abuse. I know this would not have helped in my case. And in the cases of the other survivors I know. It was our environments that led to our experiences. The adults ignored it, were in on it, or in denial.
Apple is trying to solve a complex social problem with an algorithm. One that we are supposed to trust has been made without errors or bugs.
I can’t see the upside here.
They step back on that when the problem is turned around into "why does your house have walls? why do you have locks and doors?". It's not because you have something to hide, it's because of privacy and safety. And we need these spaces to provide us with a modicum of safety and privacy because some discussions can only emerge in those spaces, some ideas need to be discussed in those safe spaces and because I wouldn't want that the world in which modern children grow up in has entirely given up their right to privacy. We always end up using the bad as something that needs to be guarded against, but we seem to fail to see all the positive experiences that have emerged in those safe private spaces as we were growing up...
Hope you are better now and thank you for the write up!
And at the end we will be allowed to talk only about the stuff that three or four corporations and their political associates are fine with.
Where are all the stories of people jailed for false positives? Or even arrested and released? There aren't any because they don't arrest people on apple/google/facebooks say, I would assume they just open an investigation. I would also take a guess that people consuming this content tend not to have a single image that might flag this system, it must be extremely unlikely for someone to trigger two or more false positives.
As for the photo scanning isn’t it already happening?
It can absolutely help with misidentifying individuals forever and ever.