If the machine says "dude is terrorist based on XYZ" and the human cannot realistically verify all of that is factually correct (perhaps the subject's phone was lost as the subject walked by a mosque?), then it is much easier for the human to say "Data says this dude is terrorist" than it is to say "Data says this dude is terrorist, but the data is probably wrong and we shouldn't..."
The existence of the data itself is a threat against every subject the data includes, at a minimum.
The issue today is that the leadership (in many areas of life from business to military to government), who make the decision to kill/censor/interrupt business/etc or not, are saying "we have to follow the data" without having any understanding of what that really means.
Ultimately, this creates false confidence both in the decision-maker and those that are following their lead. I find it unlikely that there would be anywhere near the same willingness if the 'intelligence' many of these decisions were based on didn't seem as rich and unmistakably correct as it often does.
Of course, the practical effect here is that leadership gets to blame the algorithm/model/data instead of having to accept the blame themselves. If only those pesky engineers and nerds in the lab were better at the job we'd bomb less foreigners.
I'll give you an analogy that might help. Let's say that a personal investigator is following you. They have a GPS tracker on you. They can see where you go, who you talk to, for how long, what you buy, etc. The only thing is that they don't know what you are talking to people about or exactly what you buy (but they know where you bought it from). Would you feel comfortable with this person following you around?
I'm assuming not, because I don't know anyone that has answered yes. It feels like an invasion of your personal space, right? They can still learn a lot about you and your habits by doing this, right? But all they've gathered is metadata on you. So why do you feel uncomfortable?
Goal of metadata investigation isn't to directly target you, most of the time. It's to put you in the bucket of interesting people, that government will pay attention to.
It's exactly the same as ads on the internet - they maybe classifying you as a person potentially interested in computer security because you're visiting tech crunch. Are all people visiting it interested in computer security? Of course not. But you're many orders of magnitude more likely to be interested in it, than a random person from the internet.
> Even by that account, the scale of collection brought to mind an evocative phrase from legal scholar Paul Ohm. Any information in sufficient volume, he wrote, amounted to a “database of ruin.” It held personal secrets that “if revealed, would cause more than embarrassment or shame; it would lead to serious, concrete, devastating harm.” Nearly anyone in the developed world, he wrote, “can be linked to at least one fact in a computer database that an adversary could use for blackmail, discrimination, harassment, or financial or identity theft.” Revelations of “past conduct, health, or family shame,” for example, could cost a person their marriage, career, legal residence, or physical safety.
> Mere creation of such a database, especially in secret, profoundly changed the balance of power between government and governed. This was the Dark Mirror embodied, one side of the glass transparent and the other blacked out. If the power implications do not seem convincing, try inverting the relationship in your mind: What if a small group of citizens had secret access to the telephone logs and social networks of government officials? How might that privileged knowledge affect their power to shape events? How might their interactions change if they possessed the means to humiliate and destroy the careers of the persons in power? Capability matters, always, regardless of whether it is used. An unfired gun is no less lethal before it is drawn. And in fact, in history, capabilities do not go unused in the long term. Chekhov’s famous admonition to playwrights is apt not only in drama, but in the lived experience of humankind. The gun on display in the first act—nuclear warheads, weaponized disease, Orwellian cameras tracking faces on every street—must be fired in the last. The latent power of new inventions, no matter how repellent at first, does not lie forever dormant in government armories.
take a look at the history of the behavior of intelligence services through the 20th century and ask yourself how comfortable you are with this power being wielded by anybody.
you may not feel threatened by this arrangement now, but how confident do you feel that these tools will always be controlled by people you trust?