81% of 'suspects' flagged by Met's police facial recognition technology innocent
news.sky.com
news.sky.com
Seriously though, for anyone interested in understanding this interesting statistical principle, check out the "Intuitive Explanation of Bayes Theorem" [1]
[1]: https://betterexplained.com/articles/an-intuitive-and-short-...
[1] https://en.m.wikipedia.org/wiki/List_of_busiest_London_Under...
NYC population is 8.6 million in 2017, so there are only 8~9 criminals in NYC? Wow.
Come on, you should give a much better estimate.
Once police have the right to search, enter, confiscate. and arrest probable cause is out the window.
I suppose it could be worse and they just take an aggregate face of all (accused?) criminals or for particular crimes (white collar?) and match against that... hey your skin was brown, or hey you didn't shave... you appear Male.
If yours isn't already as bad as ours, then just wait.
The iniquities of pre-trial detention and cash bail are fixable. Racially biased arrest, conviction and sentencing rates are fixable. Needlessly punitive sentences and an over-reliance on incarceration are fixable. Miscarriages of justice due to pseudoscientific forensic evidence and under-funded public defenders are fixable.
Don't make the UK out to be some paradise (legal or otherwise), it struggles with plenty of its own problems.
The UK is far from perfect, but most of the criticisms of the American criminal justice system simply don't apply.
The intentional homicide rate is substantially lower in the UK (1.2 vs 5.35 per 100,000), as is our rate of killings by law enforcement (0.2 vs 30.4 per 10,000,000). The UK may have slightly more property crime (12.2% vs 10% of population victimised), but it's not clear if that comparison is statistically valid.
https://en.wikipedia.org/wiki/List_of_countries_by_intention...
https://en.wikipedia.org/wiki/List_of_killings_by_law_enforc...
https://www.nationmaster.com/country-info/stats/Crime/Proper...
In the UK, it seems, almost everyone has a tale about when they were burgled, when they were mugged or when their car was broken into.
Is this so in other countries, I think not?
Are you a WASP? I guess you're safe in US.
Are you black, latin american, native american, foreigner? I'm quite sure you are safer outside of the US.
It still takes 2-3 years to get to trial for serious offences during which time your name and face are plastered all over the papers (at least locally).
Mounting a successful defence can cost 10s to 100s of thousands of pounds.
Fighting off charges of which you are innocent is not fun. If we assume that the UK's system is good enough to just increase the arrest rate and it will all come out in the wash this would destroy many people's lives.
Although it is clear that the cuts have significantly impacted family, immigration and welfare benefits cases, it is not clear that they have affected the fairness of the criminal legal system.
Police have always relied on facial recognition, just not facial recognition carried out by machines. We accept imperfect facial recognition from machines if it is faster, and leaves less work for humans to deal with.
I've yet to meet a customer who's looking at this technology as a way to make decisions. In every case it's there to help make their analysts' jobs easier, by sifting through the chaff and presenting them with the data that makes the best use of their skills.
Note: I work for AWS.
Around 15 years ago I would often schedule last minute flights while traveling for work. When I did, I almost certainly would receive an enhanced screening from TSA. I was the false positive, and when traveling on tight schedules, I was not amused at the frequency in which I was considered suspect (in contrast to the treatment from the airlines, who treated me like royalty). Of course, TSAs true positive rate is 0%.
There are two big assumptions you haven't made explicit. One, that the FN rate is low. If it isn't you haven't necessarily improved your SNR at all, you're just spent resources looking in the wrong place. Two, that the cost of a FP is 0. The latter obviously isn't true, so you have to weight the positive and negative impacts across all scenarios before you can tell if this is a net win.
Think of this in a different context; say I have a new cancer screening protocol, with a low FN but also a high FP rate. So everyone we mark as positive gets a follow up biopsy. You say "great, we catch x% more cancers now than the old way, good win". Until someone points out that biopsies have a y% rate of catastrophic infection resulting in death. Depending how these rates balance out, we may be killing more people that we are saving....
This is a highly black-and-white way of thinking about the implications. "Heck, it's only a problem if they get executed."
For example, roadside drug test accuracy rate is abysmal but innocent people are being put in jail because uninformed LEO either don't know or don't care baout the rate. If you can't afford to fight the false charge, your life/record/job/etc is screwed.
Also, these false positives tend to have a conveniently racist default twist in our country.
"59% of the Houston defendants in drug arrests involving false positives were black.", I expect the number that didn't fight the charge and were wrongfully convicted is quite high as well.
https://www.nytimes.com/2016/07/10/magazine/how-a-2-roadside...
I'd say a society is more successful the more they strive to minimize law enforcement contact.
Friendly reminder that our _entire_ legal system was arranged by the Founders so that innocent people wouldn’t be incarcerated even if that means some guilty people walk free.
Your reasoning flips the most important feature of our criminal justice system on its head.
I know most people on HN haven’t dealt with the criminal justice system but being wrongly charged and, if you’re lucky, beating the charges is an incredibly awful experience. Just being charged will follow you for the rest of your life.
Remember this is used by police to find suspects, not to replace the judge.
It means that just looking at a picture, 2 suspects every ten are successfully recognized
Compared to the system we have now, were to find who's guilty police have to go trough a list of suspects were all of them except one (or a few) is innocent, 2 every 10 is very good.
We're not talking about considering people guilty until proven innocent, but helping police to investigate in the right direction.
In UK almost a quarter of murder cases are still unsolved.
https://www.ibtimes.co.uk/almost-quarter-all-murders-are-sti...
It means you have to explain the arrest on any enhanced DBS you need.
It means you probably longer qualify for visa free travel to the US and will need to ask the US embassy if you have to get a visa or not.
He presents useful tools for everyday people to understand this, mostly by converting percentages to real numbers and drawing probability trees.
They declared the system, which wrongly suspected up to 600 people a day at a single train station of being wanted criminals, but missed a lot of actually wanted criminals too, a major success.
Mind you, they used high resolution photographs as the training input for the system, something police does not necessarily have available of criminals and suspects they try to track down.
https://www.ccc.de/de/updates/2018/debakel-am-suedkreuz (German)
That classifies people as "dangerous" without a trial and allows for the temporary imprisonment of those people without any court involvement whatsoever?
The government that was a driving force behind the copyright directive (aka article 13)?
The government that made unconstitutional laws to allow police to install trojans on people's devices?
That tried to push an EU-wide censorship infrastructure with secret filter lists under the guise of "fighting child porn", while lying about child porn (e.g. they claimed child porn is not illegal in India, which is not true) and where then-family minister von der Leyen (now about to become EU commission president) screened actual child porn to journalists at a press conference to influence their reporting?
The government that changed the law so that now you have to talk to the police most of the time (as a "witness")
That helped the NSA to spy on communication of other EU leaders and German politicians as well? And then tried to hide it when caught, using every dirty (legal) trick imaginable?
That regularly does mass surveillance on peaceful protesters? Or let the police lose to bash up a few of them?
That made a law that essentially turned social media companies into law enforcement agencies with no supervision?
That currently tries to create laws to force service providers to turn over communication "even if it is encrypted", aka defacto crypto backdoors?
and so on ...
Whenever you think of a foreign government as good you just have to assume you don't know enough about that country to know about all the shit they do.
But we do it with "style": e.g. our former chancellor (head of government) Schröder is a lobbyist for Russian gas and oil these days and is best buddies with Putin, to the extent that last time Putin was sworn in as president Schröder was the second person to shake Putin's hand, right after the head of the Russian-orhtodox church, but before the Prime Minister of Russia.
His vice chancellor at the time, Fischer, lobbied for a competing gas pipeline after leaving government... did I mention that Fischer belonged to the Green party?
Germany also managed to delay the ratification of the UN Convention against Corruption for about 10 years, citing "privacy issues" that could arise if you made politicians to disclose their income e.g., and finally passed a law that is essentially completely useless. Yes, really, the same politicians who were pushing for data retention of communication meta data of literally everybody in Germany cited privacy concerns...
But what if you want to donate large sums to a political party or candidate before an election or before a vote for some law you want? No, that would be bribery and corruption! So instead, just donate your large sums of not-bribes-I-swear AFTER you got what you want, and problem solved. Throw in a lobby/board of directors job or two for some politicians who needs an income supplement or hobby when retired, and you're done. No corruption to see here, please move along.
I'm still not sure what it's good for though, if the plan is to stop an interogate those 600 people this is still infeasible.
Whittling 90,000 photos down to a set of 600 is a major accomplishment. Even if 599 of those are false positives, that 600th person would probably never have been caught, because it wouldn't have been feasible to review all 90,000 photos manually.
Society needs to accept that some percentage of crimes will invariably occur. Our question is "what maximum level of crime can we accept?" which then determines the time and resources required to achieve that level. I have no patience for this "zero-tolerance" rhetoric.
I agree with you.
Said that way seems that you are given up. I think that it can be better to say "do you prefer to catch 0.5% more criminals or to reduce 0.4% deaths by cancer". Or something like that.
People usually want 100% safety, that is not possible to achieve, and to try to get it they will give away thousands of lives that could have been saved with the time and effort invested in that impossible goal.
All I'm saying is that these tools are clearly doing a useful job and pretending like false positives are an insurmountable problem is a bit silly.
Let's actually do some crude math to try to estimate how many new cops we'd need to hire in Germany just to check out those false positive hits. I mean, what's the point of such a system if you do not check out the hits and make sure, right?
Let's assume it takes police 15 mins to check out a potential hit in person (finding the person, checking their identify and asking some questions), and they work in teams of 2 to be able to secure each other.
The Deutsche Bahn alone says in 2018 they had 5.7 million passenger in Germany per day traveling by rail[1]. Let's assume all the train stations get equipped with with this tech and let's assume their 0.67% false positive rate holds. That's 38,190 checks you need to do on innocent people, per day, or 1591.25 checks per hour.
A two person team can do 60 / 15 = 4 checks per hour. So you need to have 400 such teams at any given time. A shift is only 8h not 24h and people only work 5/7 days per week, so 400 * 2 * (24/8) * 7 / 5, so you need to hire at least 3,360 police officers (well, probably more like 5000, because we completely disregarded a lot of variables, such as vacation days, etc) just to harass DB rail passengers who were incorrectly classified as threats, nationwide. Remember, those are all false positives, not one of those people is actually a criminal the system is meant to be looking for.
IIRC an average police officer costs the state about 90K per year on average in Germany. So you'll be spending 300+ million EUR per year on harassing innocent people just because your tech is kinda shitty.
Now if those 3,360 police offers would do actual useful work instead and prevented or solved some crimes...
Also, I don't want to be that person who happens to have a similar enough face to some criminal so the face detection triggers on me all the time.
[1] https://www.deutschebahn.com/resource/blob/3992284/f9331633c...
And again I don't think you're asking the right question to begin with. There is a whole separate argument to be had about whether we as a society actually want these kinds of systems and whether we think they are worth the cost (whether they require 300 or 3000 additional officers). But this system is being developed under the assumption that people do think it's worth it, and it does a pretty good job of making it a lot cheaper than it would otherwise be. I wouldn't call that a failure.
As a side-note, I would imagine this would not actually be used routinely in every single station, at least not in the beginning. You could only activate it when needed to catch very high-profile criminal. For example after the attack on the Berlin Christmas market the terrorist managed to travel all the way to Italy (?) before being stopped. If the German police could have fed his photo to this system that would have been extremely helpful, even if it would have meant a couple hundred officers having to review photos for a day.
Yes, you have no data for this. Sieving out 95% of hits just because a human glanced at the results is a ridiculously optimistic estimate, even if you manage to staff the screens with humans who are actually good with strange faces; and most humans are not. Humans on average not being good with faces (of strangers) is why witness statements are so unreliable and put so many innocent people in prison (and we have data for that). And you want to sieve out faces that the computer already thought were similar enough to warrant a match, which won't make it any easier.
They're trying to sell a horrible system that does not work correctly to the public. It doesn't matter if the public thinks it is worth it after they massively misrepresented what it actually can do. (Oh, and the German public isn't exactly known for been keen on surveillance either).
>I wouldn't call that a failure.
I would and I do.
>If the German police could have fed his photo to this system that would have been extremely helpful, even if it would have meant a couple hundred officers having to review photos for a day.
This is not backed by any data either. He could have easily been one of the 1/3 of people the system fails to detect. Or he could have further disguised himself for the system not to trigger. Oh wait, he did[1], with a beanie and a scarf and later with hood. And nobody took notice, because a lot of people looked like that as it was cold outside. In the summer, you'd just get some big ass sunglasses to cover enough of your face that the already poor system would not have triggered.
And by the time the police actually had a preliminary ID and a preliminary picture of a sufficient quality that would have allowed facial recognition, he already had left Germany. And by the time they actually had fingerprint evidence he even was in the truck, he had traveled 3 or 4 countries.
The likelihood that such a system would have helped finding Amri, even tho he actually took trains and didn't just go into hiding close by or took a car, is minuscule. Those "couple hundred officers" would most probably had more success if they spent their time actually investigating Amri and his acquaintances[2]. Also, let's not forget that by the time we would have used the facial recognition system, he had already murdered the truck driver and carried out his attack murdering another 11 people, injuring another 55.
So in the end, even your terrorist-strawman here is pretty bad at justifying such a system.
[1] https://www.spiegel.de/politik/ausland/bild-1128683-1091474.... https://www.spiegel.de/politik/ausland/bild-1128683-1089847....
[2] Well, maybe... Given the massive failure by law enforcement before the attack, I am actually not too sure. Amri had been under surveillance for months before the attack, and they even had a snitch close to him, who even drove him from NRW to Berlin. He had previously been convicted in Tunesia (in his absence because he had already fled to Italy) of robbery, and was further investigated for theft and drugs, and in Italy he was convicted of arson and battery, and spent more than three years in jail, where he started a lot of fights and was already being tracked by Italian counter-terrorism because he showed lots of signs of radicalization. He was under surveillance in Germany by counter-terrorism as well, but they further found out that besides violating the movement restrictions of his asylum, and using multiple identities (they knew about 3 or 4 before the attack, but in the aftermath they found he was using up to 14), and his links to radical Islamist groups, he also paid his bill by dealing drugs. He also was known to the US agencies and had been put on the US No Fly list. His asylum had been denied, which prompted him to try to go to Switzerland with fake documents (to marry somebody there and avoid deportation), but he was caught and jailed, but then let go again because his replacement papers from Tunesia were expected to take more than 3 months to arrive. Let go even tho he was known to be a flight risk, a violent convicted criminal, a drug dealer and very chummy with radicals, and even tho he could be jailed for more than 3 months for being a deportee alone awaiting deportation.
This guy was a walking red flag, and German law enforcement had enough evidence to put him into a prison cell for the drug dealing alone, and he had a deportation order, but they decided not to act.
Even though no one is proposing such a thing, it's basically guaranteed multiple police forces will do just that.
I know that sounds ridiculous, because it is, but essentially every technology handed off to police forces gets misused in the worst way you can imagine. Not all the time, of course, but some of the time.
I really like the idea of tasers, but unfortunately the claims that they're used as torture devices have a lot of validity.
But if 599 innocents are harrassed by the cops, then in all likelihood, the system does more harm than the crime it's seeking justice for
Another factor is that this may be challenging when starting from a facial recognition point in a crowd and trying to identify a person, but what if you're starting from a list of "the usual suspects" who've been thoroughly photographed, had gait analysis done and had their social networks mapped? What if you start from "do we see any sign that any of these 50k individuals are in the crowd?"
Marijuana is illegal, so human tracking should be illegal too.
You thought you ever had privacy?
I'm not sure I really see the connection here.
Privacy must be preserved for the same reason since we still haven’t yet lived without privacy for long enough to see the consequences.
I'm curious, are any MAC-like IDs broadcast plain-text as part of normal cell phone connections that could be passively sniffed? I must have the wrong keywords, but don't seem to be able to find anything on this.
IMEI/IMSI.
Seriously, it's amazing how well framing a conversation works to divert it.
Where are all the comments about the stasi, securing databases, corrupt law enforcement, and not collecting information in the first place? Instead we're arguing over what percentage of "innocent" people can be hassled by the total surveillance state for it to be acceptable.
There is a totally valid question of whether we want surveillance at this level, and whether it is giving too much power to the government, which may not always use it for valid or good reasons. I don't support at all the use of mass surveillance because it's too easy to abuse, but in a world where we carry a location beacon with us everywhere we go, I don't know if we lose much if there are cameras in public places?
However, those are a lot of assumptions— there’s lots of ways this could actually be as bad as it looks on the surface:
- Is the output of this system being used as evidence in a courtroom instead of a lead generator?
- Is there some systematic bias in the false positives that result in a particular class of innocent people being harassed by the police during their investigations?
- Does the system reliably include the actual suspect in its set of potential suspects?
I’ll refrain from commenting on the social acceptability of this system, as I have never been a part of UK society and thus lack a baseline against which to compare it.
^My brain is refusing to do the arithmetic tonight for some reason.
More inaccurate, sure, but I'm thinking that with the amount of government money involved it'll slowly improve. Or at least society will reach a stage where we don't care about the incorrectly flagged anymore.
A better question might be: "given how accurate this facial recognition system is, what if anything in addition to a match should be required for an officer to detain someone matched by it?".
I don't have an answer for this one. This tech is new, and complex, and hard. We'll likely abuse it before we figure out how to properly balance human rights and protecting the public.
I mean, I get that people don't like facial recognition for privacy reasons. But that seems totally orthogonal to this report, which is criticizing its efficacy, on very odd grounds.
Seems unlikely.
But with technology we can suddenly do it 24/7, numerous places, and for all sorts of crimes at the push of a button...the implications of that many people as suspects, that volume of innocent people accused seems pretty bad.
The criminal justice system always had lots of false positives that go silent and unnoticed, simply because of exposure bias (you never hear about the guy they sent to prison for life again).
That's pretty darn useful!
if it correctly finds anyone on the watch list, but they comprise only 19% of the results returned, and the police force knows it, that strikes me as a great screening tool. just need some people to sift through it.
i doubt it finds the folks on the watch list that reliably though.
How large does it have to be to be useful?
Bueller?
Bueller?
I wouldn't be surprised if these systems are used specifically in cases with weak evidence. If you had phone records or other hard evidence, you wouldn't need to rely on face matching. So the faulty system ends up used in the exact cases where it can do the most damage to innocent people.
Here's one example: https://www.manchestereveningnews.co.uk/news/greater-manches...
"Mr Scott, from Truro in Cornwall, spend nearly three months on remand in Strangeways after being charged with a single count of rape.
He strongly protested his innocence - and told detectives in interviews that he had never been to Manchester.
The Crown Prosecution Service formerly dropped the case during a hearing at Manchester Crown Court after the private firm confirmed the DNA blunder.
" Despite warnings of DNA contamination at the laboratory less than two week earlier, Adam Scott was arrested and held in custody – even though he had been hundreds of miles from the scene of the crime.
https://www.independent.co.uk/news/uk/crime/rape-accused-ada...
I suppose using colloquial language by using universal phrases for the sake of clarity is a 'no-no' because some pedant will call you out on it. How about this: "Convictions due to facial recognition false positives are low probability outliers to the extent that they will not impact the civil rights of citizenry".. Is that better? Does that improve conversation? Should we not provide a modicum of charity in interpreting the arguments of others?
BTW, you realize you undermined your point as well?
DNA is an incredible forensic tool, that outweighs any outliers. If you're claiming facial recognition is or is going to be like DNA identification, you lost your argument.
https://www.theatlantic.com/magazine/archive/2016/06/a-reaso...
From the beginning, Sutton and Adams denied any involvement. They both had alibis, and neither of them matched the profile from the victim’s original account: She’d described her assailants as short and skinny. Adams was 5 foot 11 and 180 pounds. Sutton was three inches taller and 25 pounds heavier, the captain of his high-school football team.
The DNA evidence was harder to refute. Having seen enough prime-time TV to believe that a DNA test would vindicate them, Sutton and Adams had agreed, while in custody, to provide the police with blood samples
Reality is that DNA evidence overrides actual evidence time and time again, and people are wrongfully sent to jail, have their lives ruined, and even die, because of it.
The same thing will happen with facial recognition, gait, etc.
This system performs much better than existing tools like drug dogs.
I guess we subject the crowd of innocent false-positives to that statistic unecessarily.
This technology is primarily intended for identifying suspects; an 81% false positive rate is remarkably good and extremely useful if you're scanning a train station for a terrorism suspect or a public street for a wanted fugitive. In this application, a relatively high false-positive rate is entirely tolerable (from the perspective of the justice system, at least) because checking the identity of five people to apprehend one serious criminal is a complete no-brainer.
No. This is a fully-general counterargument. You could say this of everything: 'innocent people could go to jail because of X!' This is true and irrelevant, because it is true of both good things and bad things, and not even wrong. It is sheer concern-trolling. The question is whether X works well enough. Nothing in OP or your comment remotely establishes whether it does or does not. Citing '81%' is merely throwing out scientific-sounding rhetoric. (It has numbers and it looks big, therefore it must be bad. It would be the same if it was 98.1% or 81% or 8.1%.)
So again: is 19% not enough? What is enough?
Bueller?
Bueller?
[1] There's several voluntary ID schemes for younger folks to prove they're old enough to be in a bar, buy alcohol or cigarettes etc.
The UK had them through the war and got rid in the 50s because they were thought an infringement of civil liberties - because police were asking to see them as habit, not for reasons of suspicion - funny how attitudes change. Not having ID makes something like facial recognition stations set up in the street seem an even larger overreach. We're supposed to be able to go about our business without interference unless suspected of something. OK, it doesn't always work quite that way in practice, but the principle used to be much clearer, and far better respected... I rather resent losing that.
It made for a bit of an oddity back where there were the troubles in Northern Ireland and the heavily policed border between north and south. You could get on to a ferry to Dublin from the UK with nothing. No ID, no passport, nowt. I did that several times, I don't remember anything other than a ticket check. Change some pounds into Irish pounds when you get there.
What happens to a poor sap who the facial detection system thinks looks like a known violent criminal? In the UK it means constant harassment. In the US it means eventually getting shot.
-What is the level of impact on the individuals caught up in a false positive? A minor inconvenience? A major issue?
-It takes resources to deal with false positives, even 1 in a 1000 can be a lot of people. The question isn't how much it costs though, it's how much does it cost without the automated facial match, and how much to the false positives cost, and what is the difference between them?
But if there are 1 in 10,000 people which are criminals, then 5 total people out of 10,000 will be flagged. So the chance of being flagged would be 5 in 10,000 or 1 in 2,000 which would be about a .05% false positive rate overall.
I think everyone forgets about in a situation like this is the false negative rate -- or missing a flagged person. If they miss 90% of the people they flag, then the system is pretty much worthless.
His life is going to get a lot more interesting with technology like this, since we know the false positives tend to have a specific race.
lol, I am sure the police are, we as a society however should not be allowing the police to dictate what is and is not a "success". They have a MASSIVE conflict of interest here
The police would be happy with a 1% or even less hit rate
My point is, in some cases false positives are ok, in other cases they are not.
If they wanted to spend money on something that doesn't work, they should have purchased darkpuma™ brand divination rods.
That's not to say that I have particular confidence that the system is being deployed effectively, much less ethically, but your dismissal is unsound.
If it cannot be deployed ethically (which you seem prepared to concede), then it's not fit for purpose and is therefore not a real accomplishment.
My dismissal is that the error rates make it unsuitable for real use by real police officers. So no, my dismissal is not unsound. If it hasn't already, the purpose in practice of this machine will devolve into confirming the unfounded biases of police officers harassing the innocent (e.g. ADE 651, as mentioned by another user.) In the real world, devices like this are not fit for police work. Full stop.
If "find the people on this list, with near 100% accuracy" is something we're willing to accept, I think it's possible to build a system around this technology that doesn't deserve additional objection (outside detailed analysis of cost effectiveness or whatnot).
That outputting "there's a 20% chance they're on the list" might be used to deliberately harass those who aren't on the list is an argument that you didn't make in your first comment. Certainly safeguards against such behavior would be a necessary component of appropriate use of the technology - but it doesn't seem technically or socially implausible. There's a wide margin between a system that outputs "we think this person is sketchy" and a system that outputs "we think this person might be that specific person."
All of this said, I agree that there's reason to be skeptical and keep close watch to ensure rights aren't being trampled. That's the case without this tech as well.
Between cell phones, Amazon Ring, and Nest Cam, we're there.
Voice Assistants and Cloud Connected Camera however are no where near a requirement for participation in society
That this would be allowed in a modern civil society is atrocious yet the public went along with it despite incident after incident: https://en.m.wikipedia.org/wiki/Stop-and-frisk_in_New_York_C...
Turn the machines off and the error rate will be 0%.