I don't see how it can work any other way.
If cops can get a warrant on mere hunches, then I agree we have real problems. But a separate problem.
Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this.
The designers know exactly which features are causing responses/predictions in the model, their respective perturbation sensitivities, and have clear bounds regarding adversarial inputs/outliers.
“Not knowing how the model works” is definitely true for deep learning, though.
First of all, deep learning absolutely is used in mission critical systems: https://www.techrepublic.com/article/intel-and-ge-healthcare...
Second, simply because designers use systems that are formally interpretable, that doesn't mean that designers know exactly what causes a given response in the model. Formal interpretability means you know what function the model is approximating -- there's a long way from that to a human-readable interpretation.
Finally, even if the designers can tell you what features cause a certain response in the model, they can only tell you that for features that are encoded in the data! There are plenty of features that aren't encoded in your data that can nevertheless affect the outcome of a model!
I don't know much about ML. Are you saying that we don't know how to interpret any ML models at all?
Regarding your final point, isn't that true with or without ML? Any mission-critical design process should scrutinize the solution to see if it's complete enough and correct enough.
Deep learning models are black box models, which means they are not formally interpretable. You can still sometimes get interpretations out of these models using various methods, but they are fairly underdeveloped, and without actual theories of neural network behavior I don't see that improving anytime soon.
You're right, it's true with or without ML. In fact, it's true with human-run systems, too. Consider a police officer who is more likely to pull people over in a certain neighborhood, and that neighborhood was 95% AfAm. IF they were asked why they pulled over more people in that neighborhood, they could say they were discriminating against the neighborhood, which, in and of itself is not racial discrimination. Of course, further inspection of that police officer's records could show that they are biased towards neighborhoods with a high AfAm population, which would be racial discrimination.
The same scenario can easily arise in an ML context, but interrogating a machine is a very different context from interrogating a human. First of all, people believe that computers are innately 'unbiased' because they are computers, so making the case that an algorithm is biased is already more difficult. Second, going back to the point I made before -- interpreting a model is not the same as providing a human-readable explanation. Asking a question about racial bias in a model which doesn't even encode for race (as many do, in an ill-conceived attempt to be 'neutral') requires skilled people to understand how to ask the question and how to interpret the answer. There's no plug and play process that one can follow to "scrutinize the solution to see if it's complete enough and correct enough".
This is how it functions at first.
Fast forward a few decades and you'll have doctors saying "The computer model flagged this spot as a concern. Our human review can't find anything, but we know that studies show the computer model has a 95% confidence in locating problems so we recommend surgery anyway. Surgery is lower risk than assuming the model is wrong."
Tool dependency changes over time.
Judges cannot just sign off on warrants and sentencing just because "the computer says so", for all we know the computer may be programmed to say "poor and non-comformist = guilty" or some other nonsense, and/or reprogrammed after every election. We need to be able to trust this stuff.
Is there any source that confirms the belief about judges deciding based on computer output in the past or in the future - relevant cases, law, etc?
First the Sheriff’s Office generates lists of people it considers likely to break the law, based on arrest histories, unspecified intelligence and arbitrary decisions by police analysts.
Then it sends deputies to find and interrogate anyone whose name appears, often without probable cause, a search warrant or evidence of a specific crime.
They swarm homes in the middle of the night, waking families and embarrassing people in front of their neighbors. They write tickets for missing mailbox numbers and overgrown grass, saddling residents with court dates and fines. They come again and again, making arrests for any reason they can.
One former deputy described the directive like this: “Make their lives miserable until they move or sue.”
It seems that police departments are happy to buy tools for stuff like this, and private companies are happy to make a buck. But are these systems vetted by anyone on behalf of the public? Some like sting rays and breathalyzers have been hidden behind non-disclosure agreements, kept out of open court, etc.
Other than the above I've also heard of computer systems in NJ advising judges whether a suspect is a risk of not appearing in court (as they are reforming the bail system). I wondered if the criteria is published, how it reviewed and modified, etc.
I'm just concerned about the increasing influence of hidden algorithms on our society, and very concerned in general that the government is going to hook all of its databases together and do more of this, a la Chinese social scores, etc.
[0] https://www.techdirt.com/articles/20200907/12212945257/flori...
Are you speaking from experience or from intuition?
We are not talking about black box models here. "Phones that pinged near the Capitol" is a very specific query. They want to identify those who trespassed and those who aided the trespassers. I believe it's pretty fair.
Getting a list of suspects is rarely a problem for law enforcement; the difficulty is in winnowing it down to the actual culprits. When a body is found, for example, family and acquaintances are all initially suspects, and experience has shown that summarily dismissing any of them, merely on intuitive grounds, will eliminate some fraction of actual culprits.
If a system did start suspecting the actual culprits with a significantly higher success rate than people achieve, there would be much reason to reverse-engineer the process in order to figure out how this was accomplished, as doing so would provide clues (and, ultimately, evidence) that otherwise could only be found by an independent process.
This assumes that due process exists, such that unsupported accusations are not taken as evidence, but if due process has been abandoned, we would have a much greater problem than that posited here.
No. They will come up with some fancy sounding term to describe these situations. Some experts will agree this is a sound method and it will be used to get warrants.
They will just put the cause as "suspect by unattended abnormalities" (aka phone not used during crime)
Everybody will smile. Warrants will be made. Life's turned up side down. And likely false arrest and convictions.
Even if we program rules into th3 AI to avoid this there is a real chance that the AI could work around and add people to thr list that it suspects is involved in one crime but could not list thr suspect do to rules, but a new crime opened a way for it to allow a suspect of crime one to br listed on crime 2.
Law enforcement doesn't want a system that incorrectly flags hundreds of people, unlike what some people seem to think. They want systems that reliably flag potential suspects, because that reduces work in stead of increasing it.
However, there is a long history of wrongful convictions and police and prosecutors using bad data to get them. Check out the Central Park Five for a big name one.
Precision data gives them convictions and convictions give them promotions. As they say, any metric that becomes a target... This is why giving the police invasive surveillance tech is a terrible idea, they will only focus on the data that fits their narrative and discard the rest. The defense doesn't even need to know that any other data exists.
If your system produces data points with 8 significant digits of wrong data, we have two problems: the system and whoever approved the purchase.
Any convictions will have to be corroborated by other evidence.
Scary times.
https://www.theverge.com/2020/3/7/21169533/florida-google-ru...
This whole line of "what could go wrong!?" starts from a much deeper sickness in the way Americans view the criminal justice system.
Proving one's innocence once the powers that be have determined one to be guilty will cost hundreds of thousands of dollars, will almost certainly mean loss of job, family and friends, etc. And you might still go to prison for the rest of your life anyway.
The sickness is in how the criminal justice system works. Jury trials are a rare edgecase, most detminations of guilt are made by DAs. For most people in this country, being arrested and charged means you will be pushed into a plea deal and those that fight for their innocence face retaliatory harsh charges and sentencing.
- Sitting in a parked car in a suburban street for too long.
- Going shopping at 2am.
- Walking under a bridge at night.
It's OK as long as they quickly realize you're harmless and leave you alone.
Currently.
It seems to me they harass anyone they perceive to be abnormal.
As a delightfully abnormal human, I'm very glad I was born just north of the 49th parallel. It's not perfect, but I do feel free to be myself.
I even wrote about it recently: https://sneak.berlin/20200628/the-problem-with-police-in-ame...
Police do in fact exist to enforce the law. It really is that simple.
Police exist to "keep the peace." The question is whose peace are they keeping. Well I suppose the reaction of police at the Jan. 6 insurrection vs that of the BLM protests tells us a little. Or the union busting police did in the past. Or blocking the polls for minorities.
But it absolutely isn't "that simple."
They shot a woman in the neck. What’s the difference you’re trying to portray?
I've had two illegal stops in the past year because in one I took a swig from a soda bottle going by a speed trap and the cop wanted to see if it was alcohol. The other, a cop in an unmarked car tried to run me out of my lane to follow a speeder.
Some years ago I had to deal with a small town gumshoe in Indiana who had staked out an intersection of IN route 55. This was late at night in pitch black darkness. Indiana highway signs look like speed limit signs from a distance and IN55 in particular is hard to distinguish from the 55MPH speed limit signs. The local cops know this and abuse their power to make illegal stops.
https://www.google.com/maps/@40.2361263,-87.2433936,3a,75y,1...
Indiana requires signals 200 ft from an intersection in rural areas rather than the normal 100 ft. The county couldn't be bothered to have an advance sign for the intersection so I didn't know when to signal for the turn. He and I were the only two people on the road but clearly I was a major threat to public safety.
Me and my buddy were standing on the sidewalk of a bridge, making a time lapse. Within one hour, someone called the police on us twice.
Do that 10 more times on 10 different bridges, and I’ll be shocked if that happens even one more time.
I should have said that my interaction with the US society has been that people tend to meddle with others' business more often than in other countries that I've been to.
But, yeah, I have many more anecdotes.
Likely it would happen another 8-10 times. Happens all the time, the bigger the bridge, the more likely.
Get away from the wealthy areas on the coasts and "noneya business" quickly becomes the societal norm.
Only places rich and population dense do the police have the spare shits to give about what other people are doing.
It’s important not to confuse media narratives and cherry picked videos with trends. It’s also important not to generalize across such a diverse set of individuals and jurisdictions. You’ve got everything from LAPD to an elected backwoodsy county Sheriff.
Heck, I’ve been stopped in NYC for “walking with a limp” (bike accident, fractured knee)...while wearing a $500 shirt and designer suit.
What you refer to as “cherry picked videos” are just the instances that happen to have been filmed. Seriously, after Central Park Karen and Minnesota, can we at least acknowledge there might be an issue?!
More importantly, parent wasn’t even commenting about race.
Did you try to make a joke ? Both of those are racist to the core. They are the same.
> It’s also important not to generalize across such a diverse set of individuals and jurisdictions.
Pick one.
It's really no different in banks but they are regulated and must provide proper reasoning to denied customers for the models they use in my country. It's not enough to say "your score is too low".
It's very easy to create artificial stupidity!
If we wanted to stop this, the time to complain was years ago when the practice was started in the drug war. Or even recently, when the practice was employed during the BLM riots. Any attempt to stop it now brings howls of racism. Causing police departments and other law enforcement agencies to double down on insisting that they use it on everyone in their attempts to prove the people screaming racism wrong.
The people doing the violating are going to double down. They don't want their critics being proved right. They don't want to be accused of being hypocritical. So what are they going to do?
As privacy activists, we need to make it easy for potential partners to cooperate with us. Right now, we're making it very difficult for potential partners to cooperate with us. We're putting potential partners in very difficult positions, and then asking why they won't support us?
It's kind of difficult when the privacy violations in question begin in secret. Consider police trying not to disclose their use of Stingrays, for example.
I somehow doubt police departments and intelligence agencies are going to agree to run all future uses of tech by a privacy watchdog, so how do you suggest getting ahead of the problem?
I don't have strong views on the right policy outcome, but it is not accurate to call this issue well litigated and/or settled law.
Just yesterday, NYTimes ran an article about DIA claiming a "commercial availability" exception to the only Supreme Court case addressing cell phone location data (Carpenter). If that is indeed DIA's rational, they are going to have some problems. For example, it is unlawful for the state to use commercially available thermal optics to surveil the interior of a dwelling without a warrant. I think DIA may be relying on dicta from Kyllo about devices in "common use", but their rational is secret so we won't know until it is... litigated.
[1] https://www.nytimes.com/2021/01/22/us/politics/dia-surveilla...
Presumably, this statement means that litigation will necessarily reveal the rational by presenting it.
There is a low tech solution to every problem.
For a real world example: https://en.wikipedia.org/wiki/Operation_Igloo_White#Conclusi...
Becoming a suspect shouldn't be a scary thing to avoid. You should be able to just ignore your status and wait for the police to exclude you. But somehow it's a problem in America that people just accept.
"there's nothing wrong with the death penalty if you can trust the legal system implicitly" Neil Gaiman, American Gods