A new way police are surveilling: Calculating threat ‘score’ | Hacker News Reader

A new way police are surveilling: Calculating threat ‘score’

washingtonpost.com

65 pointsdanso70 comments

>Fresno Police Chief Jerry Dyer said officers are often working on scant or even inaccurate information when they respond to calls, so Beware and the Real Time Crime Center give them a sense of what may be behind the next door.

This assumes that the information present on the Internet is accurate, a scary thought. How difficult would it be for someone to make a profile with your first and last name on any or all of the major social networks, painting you as a gun nut, a recovering drug addict, or worse?

Allowing this system to inform police work without the least bit of public oversight is irresponsible. But that won't become a concern, I suspect, until something unfortunate happens. For now, the only stories we have about this technology's use point to the 'bad guys' being caught.

> But that won't become a concern, I suspect, until something unfortunate happens.

And then it will just be dismissed as an unfortunate side effect, shadowed by all the times the system worked and did good. After all, isn't it worthwhile to have a SWAT team bust down a few doors and maybe kill one innocent person if it helps catch hundreds of 'bad guys'?

(/sarcasm, in case it isn't obvious)

I think the idea of someone fabricating legal-but-indicitave online information about you so that you turn red in the police's system and then setting you up to be considered a threat in an incident where the police assume you are dangerous but you are not and are goaded into shooting you is kind of a stretch. Maybe a good plot for a movie.

But the program itself is somewhat troubling. But right now we have a profiling system the police use. It's 'dark skin -> almost certainly has a gun' instead of 'BEWARE says green BEWARE says red'. Is looking at public info with an explicit review-able algorithm better or worse than letting officers profile naturally?

I'd agree using data like this for police purposes can be fraught with issues and is troubling but even as a privacy advocate this system seems at the very least not worse than what we currently do. Not because it is good, but because what we currently do is so bad.

It'd be a pretty easy way to get rid of someone you don't like. If you use tor and public wifi it is pretty unlikely they'll track it to you and you could set them up to get beat up/killed by police by making them look super dangerous and then calling in a tip.

There are people motivated enough to do things like this.

Remember that SWATting is a thing.

No really, how many people online get Swatted? Now if someone goes to the extra trouble, you could end up dead.

the idea of someone fabricating legal-but-indicitave online information about you so that you turn red in the police's system and then setting you up to be considered a threat in an incident where the police assume you are dangerous but you are not and are goaded into shooting you is kind of a stretch

SWATting is doing exactly that, with just a voice call. http://nypost.com/2014/09/15/online-gamers-use-police-hoax-s...

The notion of people seeing a comment they thought was sexist and hounding your employer to ensure you were fired seems kinda far fetch as well. The issue is that there are enough people out there that even far fetch scenarios will happen from time to time.

And it only seems far fetched until it happens to you. Then it's front and center.

A movie like this already exists. It's called Minority Report.

The scariest part is that this kind of program drops all pretense and explicitly characterizes all interactions with the authorities in the context of the "threat" each of us poses. It anchors the discussion in terms inherently antagonistic relationship, and normalizes aggressive policing behavior. Why aren't more people scared by this?

Although it is inconvenient for law enforcement, the government should not conduct persistent, universal surveillance on the citizenry - even if it only consists of monitoring & compiling publicly available information. The value of these tools is dubious, and the potential for abuse is high; there is no accountability for the costs in comparison to the utility, and they are routinely established in the absence of oversight by design.

Mechanisms already exist for targeted surveillance when authorities have probable cause. Normalizing casual surveillance with no specific intent marginalizes the discussion of more invasive surveillance techniques, and emboldens authorities who dismiss privacy as a trivial impediment to security.

Thought experiment: is this very different from police targeting high-risk neighbourhoods?

Police have been over-policing black and poor neighbourhoods for decades but no one is protesting that. There will certainly be a correlation between "interactions with the authorities" <-> where police spend their time patrolling, skewing the data.

The only time I've ever been stopped and interrogated randomly, simply for walking down the sidewalk with a hoodie on (in winter) in a bad neighbourhood, was visiting a friend in a housing project. Asking where I was going and who am I meeting.

All of his family and friends get harassed by police living there, despite having zero background of criminality. And in Toronto they record your name/location each time you get stopped and carded in an "anti-gang" database.

Police have been over-policing black and poor neighbourhoods for decades but no one is protesting that.

Er, apart from all those Black Lives Matter protesters that keep getting in the news?

Apologies I wasn't aware over-policing black neighbourhoods was one of their issues.

It seems to be rare that this issue specifically gets raised. Usually the media focuses on the conduct on officers upon an altercation rather than how often police are engaging with black people needlessly.

As the parent comment pointed out, the use of counting the amount of altercations as crime data is a worrying point. Amplifying the side-effects of over-policing certain areas.

A few considerations:

1) If these are good ideas and there is nothing wrong with them, than why does law enforcement keep them secret? Why don't they advertise to citizens: 'We track everything you do, including who you call, where you go, what you say online, and then we predict if you are a threat or not and respond appropriately. This makes you and the community safer!'

2) Does being black or Muslim, having far-left or far-right political views, increase or decrease your threat score?

3) Law enforcement has a very well-established history of using their power not for public safety, but to abuse and oppress vulnerable classes of people and people politically unpopular with the state (e.g., peace protesters). Just think of all the shootings, arrests, COINTELPRO, etc. etc. Why would this power be used any differently?

For example, many blacks already experience harassmet from law enforcement, including such things as pulling them over for 'driving while black', suspecting them of criminality for being in a predominently white shopping area, arresting them for no cause, etc. What will life be like for blacks on a police threat list? Will their life be constant harassment? Will the predicted threat be self-fulfilling because they will be arrested inevitably? Will corrupt police just plant something on citizens on the threat list and arrest them to preemptively get them off the street (and then tout the accuracy and value of the threat list)?

4) Police already are given the authority to shoot people based on suspicion or a feeling of threat. For example, someone putting their hand in their pocket or reaching for something in their car could be reaching for a gun. Will being on the threat list be justification for violence against citizens?

5) Aren't we presumed innocent until proven guilty? Aren't we to be judged by our actions and not what someone else thinks we will do? What happened to the land of opportunity?

> Does being black or Muslim, having far-left or far-right political views, increase or decrease your threat score?

One of the things they explicitly mentioned was "gang associations". In my US experience, groups of black or Hispanic* youths who sometimes do illegal things (shoplifting, trespassing, underage drinking, pot smoking) are often called "gangs" in cases where groups of white or Asian youths wouldn't, even if they were doing the same things. So I think it's inevitable that this kind of Citizenship Score will reproduce and fortify existing structures of racist discrimination.

* This primarily applies in parts of the US where the local system of racism classifies Hispanic people as nonwhite.

Since when has "gang" excluded "white"? The Chicago Outfit, Hell's Angels?

He didn't say 'gang' excluded 'white', did he?

He said that a group of black youths that sometimes do shady stuff will be labelled a 'gang', while a group of white youths will not.

I guess, anecdote vs. anecdote, I've just never seen that happen.

Parent comment's point is that a group of white kids that hit a mailbox are called "miscreants", while a group of black kids who hit a mailbox are called a "gang".

White gangs exist; it's just easy to call a group of black kids a "gang" because that's our image of a gang.

Turning the profiles and prejudices of police into an explicit metric, whose code can be examined and debated, seems like an idea that might have not have legs in a free society.

However, surprisingly unmentioned here is the inevitability, so long as the client is the state, that these metrics include political dissent as a threat model. Has anyone asked Beware for comment on this?

This is probably the unstated purpose of the software rather than providing stratagems for officers responding to incidents-- the repertoire of possible moves contains only "violence" or "ultraviolence", so information on which to use when responding to an incident isn't that useful.

The police already build dossiers on people who attend protests so that they will have targets to discredit or harass.

When do the citizens start building dossiers on the police?

I'm definitely for scoring the police if they insist on rolling this out. Police should not be above the law.

Unfortunately, this is not likely to happen just like LEO are able to hide their home address because safety but you Joe citizen do not have that option.

If political dissent is correlated to violence, why shouldn't it be a parameter in the threat model?

Everyone, including police, form threat models in their own minds all the time. Ever avoided the hoodied skinhead hiding his hands in his pockets in a dark alley? Threat model.

The model isn't grounds for prosecution, but it is reasonable grounds to adjust your response.

If political dissent is correlated to violence, why shouldn't it be a parameter in the threat model?

Because we know this will be implemented in a partisan way. "Dissent" implies that there is a single politics from which everything different must be considered suspect. Simply using the word implies totalitarianism.

You can see this in the US where the white people talking about the violent overthrow of the government somehow aren't considered terrorists and are relatively unmolested even if they actually occupy Federal land and buildings, but a nonwhite kid holding a toy gun gets shot in a playground.

Edit: just to drive the point home, does the threat model include a database of gun owners? Do we think gun ownership might be correlated with crime? Or is it illegal to investigate that as it is for the CDC?

How does "dissent" imply totalitarianism? If someone was to believe that women should not be allowed to vote today in the USA, they would be dissenters. Does that mean the USA is totalitarian?

As far as I can tell, they're relatively unmolested because:

http://imgur.com/AYNRxlo

> political dissent is correlated to violence

In some important ways, this is already a non-sequitur.

First, let's stipulate that dissent is positively correlated to violence, although I don't know how exactly you'd measure that. I suspect that, for reasonable measurements, it's not and that in fact it's negatively correlated. But let's stipulate.

To use the example of the current political configuration of the United States: the government is ruthlessly violent, not only within its borders but around the world.

So, which is more strongly "correlated" to violence, support for the government or dissent? If you hobnob with government officials, does this make you more closely connected to violence than if you associate with dissenters?

This is interesting since, really, cops' brains are already creating "threat scores", it's just that they aren't available to look at or tinker with how they're calculated. It's likely these computer models are inferior to the mental models, but making them explicit is a big step.

It would be a big step if those models were under public scrutiny and not a "trade secret" of the for profit company that makes it.

Fair point. But it's now also possible to lobby to get access to it

As soon as the model were public it would be useless. The first thing an aspiring criminal or terrorist would do would be study the model and then do whatever necessary to make his or her own threat score low.

https://xkcd.com/810/

Heh. I was about to write "lower their threat score by not being gun nuts and not committing felonies?" but it appears XKCD got there first...

Ah yes, more propaganda supporting a surveillance state. What makes these sorts of things difficult to argue about is all a matter of how they get framed.

"It's good for a more peaceful society. You aren't violent? Right? You have nothing to worry about then! This technology will help keep you safe. You want to be safe? Right?"

I'm glad to see many citizens are at least sharing their voice against this. But it reads all to much like Fresno took "big strides too fast" and if they had taken baby-steps there would have been less outrage. You have to whittle down people's liberties - not try and take them all at once! If you do it slowly enough, they don't even notice.

As an aside... typo + how do they know their score? I thought only operators knew it? How does she know her threat level was elevated because she tweeted about "Rage"?

>Once council member referred to a local media report saying that a woman’s threat level was elevated because she was tweeting about a card game titled “Rage,” which could be a keyword in Beware’s assessment of social media.

Why shouldn't we use statistical inference to deliver information that enables responders to be better prepared to deal with a situation? I would hate to be a false positive, but I'd rather have the chance of being flagged as "green" when police are headed my way, rather than always being presumed to be a "red" as seems to be the case now. This seems like it would be the opposite of being presumed guilty until proven innocent, for most people.

And this isn't surveillance: they aren't collecting new information, just putting available data sources together. Unless deducing things from existing data is somehow surveillance?

The issue is judging everyone before a trial. i.e. the opposite of the presumption of innocence PLUS huge scale. Like mass tracking of people using phones versus being tailed.

The anecdote from the article strikes me as a really good example of why this sort of statistical profiling is unnecessary. Guy threatens his girlfriend, algorithm flags him as red, so they... call in a negotiator and successfully de-escalate the situation. I'm struggling to see how the system flagging him as red had anything to do with their actions. It was just a case of cool heads doing good police work.

Cops already have a lot of contact with parole inmates who have dramatically reduced rights. So they'll just lump "red" in with the parole inmates. You'll hear cops saying things to each other like "he wouldn't be red unless he deserved it". Generally, having the cops prejudiced against someone before they even meet them is not going to turn out well.

US cops are not trained to prioritize descalatio in general. That's why you have that cop in McKinney doing a diving roll like Farva.

We should not use statistical inference because statistics do not always paint an accurate picture.

An extreme example is that many criminals enjoy ice cream; obviously, it's a fallacy to say that all ice cream eaters are therefore criminals. But what about correlations between crime and race, sexual preference, location? If my neighbor is a terrorist, am I his accomplice? Is everyone who lives in a bad neighborhood a crook, or are some of them trying to scrape by legally?

It seems like this argument appears on every single thread about surveillance. Data does not lie, therefore we cannot lose if we use data to solve crimes. The problem, friend, is that data does not lie--humans do. We lie to ourselves all the time, because the patterns fit and we are creatures that survive by pattern recognition. We see patterns even where they aren't patterns.

It's also very much possible to draw the wrong conclusions as you mentioned. So while data may not lie, there certainly are plenty of faulty interpretations.

What scares me is the fact the data aggregation is being done with mere promises of "trust us, we won't look" . Systems built this way are inherently vulnerable to secondary and tertiary usage. That's why I think there is a strong case for end to end encryption so that there is separation but ultimately this is still a problem because there is no way to enforce metadata constraints on data. Someone may design such a system but others will just resort to using something else. Frustrating.

Wouldn't it be better for the police not to presume people to be 'red'?

For example, here in Holland the police are (still) very much public servants alongside 'enforcers of the law'. I actively seek them out if I need assistance of some kind. My experience in Brazil and the USA, on the other hand, was that the police appeared threatening (uniforms, stance, weapons) and the few interactions I had with them were scary and antagonistic. It was my first real experience being scared of the police, and a real eye opener.

Note that there's not just shoot to kill response.

They specifically mention "property records". For me, that works out pretty well, I have enough money to afford enough property to indicate I can easily afford legal representation and will be treated as such by the police, yes mr vlm, no mr vlm, you are free to go mr vlm, have a nice day mr vlm. On the other hand, people verified to be dirt poor can expect dramatically different treatment knowing they cannot afford legal representation. On the downside I can expect no mercy WRT ticket fees, because they'll know I can afford it. You can envision traffic cameras lighting up when I drive thru a speed trap village, between my age, clean criminal record, and property ownership they'll know they can collect quite a toll off me at zero personal risk.

Aside from property, anyone who's ever posted anything vaguely drug or alcohol related on the internet, ever, now has perpetual probable cause for a breathalyzer test or a drug test down at the hospital. This is perfect for police harassment.

They must have read this article https://www.aclu.org/blog/free-future/chinas-nightmarish-cit... and misread "warning" as "inspiration"

It appears that they are only using it as a caution to ensure that the responding officers have the resources so that the public, and the offices are not in harm. Now as far as the aggregation of publicly available posts to the internet, as well as corresponding that to private-state information (previous records / offences). Its nice that SAAS today can be updated on the fly to ensure that the threat scores can become more refined, so in the example of the Yellow House score for the council member, might have been avoided.... unless hes actually a yellow based on his web traffic.....

I wonder if they apply this to pulling people over as well. Think, is the car reported stolen (alert: RED), if not then the highest color for the house hold.

But lets also keep in mind that Fresno Ca, is not Palo Alto, Ca.

So... Here's my thoughts on it. At first blush how dare they! But as I let this stew for a while and turned it over a few times, I'm arriving at a different conclusion, helped along by some of the insightful comments here.

Since the police really are doing this kind of thing in their heads anyways, what I see here is an opportunity for accountability. Did the man in body armor with the gun consult the magic number before he shot the kid on the playground with a toy gun? Or did he just get excited? Like body cams, I think this is actually something that could be used in favor of a more responsible police force than not.

Technology moves too fast for the law. Even if police believe using a particular technology is unconstitutional, they are likely to use it anyway, because it will likely take years for a challenge to be brought to court, and definitely years to get to the highest (influential) courts. Even after all this, the police will only have to stop the unconstitutional activity, and immunity will handle the rest.

States need to enact laws that will prevent police departments from using any new software without prior approval of their constituents (e.g. city council approval).

You know I hate to admit it, but as techies we are responsible for most of this power that the police have. We have created all this technology, and then some business has gone into local police departments and pitched the idea.

Kind of sad to realize there are people in our industry who don't mind doing bad things with their talents or are at best indifferent to what their company is doing. It's the "I just work here" attitude so you don't have to care if the company has an actual business model, helps people, creates a good product, or is even doing something ethical or not.

Reminds me exactly of the intro from „Person of interest“ https://www.youtube.com/watch?v=WOnQ8CD3v4g

It also is the central plot point to the anime Psycho Pass.

(off-topic but the watch in Psycho Pass was what I hoped the Apple Watch would be)

I'll also point out that this is inherently unfair fron the standpoint using a person's name as the primary key for lookup. Given the roaring success that is the no fly list (/s), people's names are not evenly distributed, there are a lot more John Smiths than say Larry Ellison. It should not mean one person is easier to identify because of their name. If that's the case, there should be an even name distribution to give everyone a fair shake. Let's call everyone Bob!

Also, police in at least some cities use systems that predict whether someone will become a criminal and give that person extra attention.

Here's one story on it that I found with a quick search:

http://www.nytimes.com/2015/09/25/us/police-program-aims-to-...

Wow, they're even calling it "Predictive Policing". Here's my prediction: By 2020, we will have fully automated arrest warrants being served and executed based solely on big data analysis, and the legality of that practice will be working its way through the courts.

Precog Unit.

The scary thing is, this is pretty much the plot for Captain America: The Winter Soldier. http://www.imdb.com/title/tt1843866/synopsis?ref_=tt_stry_pl

Oh wow, they probably just gave equifax a truly horrible idea.

I wonder if this is even legal since it creates prejudice, probably is through some loophole.

Did you ever protest a war? Ah you have a terrible threat score.

Next they open it up to employers.

Imagine that this is just employing machine learning, where the training set is police reports filled out after the fact. Is this just big data being applied to threat prediction?

Reminds me of the Sesame Score that was being talked about so much a few weeks ago.

The Mark of the Beast looks more and more real every day it seems.

It's like a high-tech version of profiling.

It reminds me of Psycho-Pass.

"Psycho-Pass is set in 2113. The Sibyl System (シビュラシステム Shibyura Shisutemu?) is actively measuring the populace's mental states, personalities, and the probability that individuals will commit crimes, using a "cymatic scan" of the brain. The resulting assessment is called a Psycho-Pass (サイコパス Saikopasu?). When the probability of a person engaging in crimes measured by the Crime Coefficient (犯罪係数 Hanzaikeisū?) index exceeds a certain level in an individual, he or she is pursued, apprehended, and killed if necessary."

Taken from: https://en.wikipedia.org/wiki/Psycho-Pass

As plausible as Minority Report. Even if possible it would not be implemented because of how the majority of people think of human free will.

I also was reminded of Psycho-Pass.

Except this isn't a brain scan in anime. It's basically a mathematical model based on prior criminal history, gang affiliation, etc.

"The program scoured billions of data points, including arrest reports, property records, commercial databases, deep Web searches and the man’s social- media postings."

Does "deep Web searches" mean the program has access to your tor activity? How is that information acquired?

No, the "deep Web" is things like friends-of-friends Facebook posts, Twitter posts to locked accounts, and friends-locked LiveJournal posts — things that are in some sense "on the web" but that a standard search engine can't find. You're thinking of the "dark Web", which includes things like onion sites. Your confusion is not unusual; https://en.wikipedia.org/wiki/Dark_web says, "The dark web has often been confused with the deep web, the parts of the web not searched by search engines. This confusion dates to at least 2009."

Reply on news.ycombinator.com