Man Wrongfully Arrested by Facial Recognition Tells Congress His Story
vice.com
vice.com
This is not really a personal complaint. My situation is still quite privileged. But I think it's an example of just how much of a techno dystopia we're living in.
It should be a personal complaint. You had precisely 0 privilege.
I don't have a problem with this per se, and at times it's even a good exercise from an intellectual standpoint, but it's at about a 12/10 at the moment.
Technically, a previously unknown near-earth object could wipe out most life on earth at lunchtime. Odds are, I'll still have that 2 PM meeting, though.
Did you know CBP agents are trained to get you emotional, piss you off, and treat you like shit? It’s an effective and quick way to get someone on edge and maybe reveal something to be investigated. Nervous behavior. CBP officers go through therapy to deal with this.
Fight fire with fire. I would make it clear (respectfully) the situation is ridiculous and push for resolution in my favor.
I was once asked for a second photo ID at the border and I didn’t have one. The officer found this very suspicious. I did get kind of pissed and asked him what kind of idiot takes every photo id they have abroad. What if it’s lost or stolen? I was waived through.
So one time we're flying domestic as a group to set up some equipment for a research project. He gets pulled aside by a couple of white TSA agents as part of a "random inspection", out of a line of about 20 people, mostly white or light complected.
He starts getting questioned about where he's flying, why, etc. Then they ask to see his ID and boarding pass. Hands them his passport, state DL, and boarding pass, and they start going off on him as to why he's got his passport for a domestic flight, how that looks suspicious and such.
He responds with "Well, I get stopped for looking Arab every time I fly, so this is a much easier way to prove I'm a US citizen." They stop for a moment and look around at everyone watching, tell him to have a good day and walk away. I'll never forget that encounter.
Privilege is real, a little anecdote.
I'm black, I've had quite my shares of DWB stops and bad cop experiences (I detailed some of them on Twitter and in this subsequent interview). That Twitter thread blew up because so many black people chimed in (from college students to famous journalists to famous musicians) detailing their ridiculous experiences with cops.
https://www.boredpanda.com/ridiculous-reason-black-people-st...
One of the more "interesting" one was I was stopped by a cop in a suburb of Boston, I think it was Newton or Newtonville). The reason given was that "I was adhering too strictly to the rules" which made me suspicious. This was a read that had a 35 MPH limit and I was strictly observing that limit. So the fact that I was following the law made me suspicious. But the reason I was following the law (even when other drivers were speeding) was because cops will use any ridiculous reason to pull me over. Can't win for losing.
Privilege is real and I see it every time I compare how I get treated with how my white friends (who share the same socio-economic profile and other markers to me) are treated. I dumped a white acquaintance I was giving a lift to Manchester NH on the side of the road when he pulled out a joint as I was driving and got belligerent with me when I got mad at him for endangering me and refused to put it out or throw it out. He just could not get it through his head why I, a black person at much higher risk of getting stopped by cops would take umbrage at his stupidity.
I'm 47. I have never done weed, not because I think there is anything wrong with it, but because given how often I got stopped by cops (driving or even simply walking), all it would have taken was for me to have a joint on me for it to ruin my life. My white friends had no such fear.
When I was younger and in college I used to work security, and sometimes I'd do security for concert venues. I saw how the cops treated predominantly black patrons vs white patrons. With black patrons, the cops felt their role was to intimidate and they tended to escalate situations and just generally be rude and aggressive. With white patrons, they went super easy (I worked security at the Fleet Center in Boston for a Fish New Years Concert (1995 or 1996 I think) (the band put on an amazing show and made me a fan of theirs that night)). White concert goers were walking around openly smoking pot (I drove home in the morning with my uniform reeking of weed) and the only things the cops would do was to occasionally ask patrons to put them out. Totally different from when the crowd was predominantly black (no black person in Boston at that time their right mind would even THINK! of lighing up a joint in front of a cop).
The facial recognition issue is particulary vexing/scary for me (Netflix has a great documentary called "Coded Bias" about this) because there are 2 things that are true 1: Facial recognition systems are incredibly terrible at telling black faces apart 2: Any rule or perceived rule violation is going to be enforced much more harshly against POC. There is wiggle room and entreaties about being mis-identified are likely to fall on deaf years
A good example of this confluence is the black girl who was put out of skating rink (at night, thereby putting her in danger) because the facial recognition system mis-identified her. The only saving grace is the venue did not call the cops (I'm pretty sure that things would have gone badly for the teenager) https://www.newsweek.com/black-teen-barred-skating-rink-afte...
Are these fishing expeditions a demonstrably effective way to prevent terrorism/trafficking/whatever?
Obligatory glad you made it out, but seriously- had any one thing gone wrong, who knows what could have happened in the end.
I used to sit and wonder about the future. As in how wonderful it will be and what amazing things we will accomplish. I used to fear dying because I wouldn't get to see any of it, much less the end of the story. Somewhere along the way I began to fear the future, to the point I didn't want to have children, I felt it unethical to knowingly bring an innocent life into that kind of terror.
Professor Farnsworth said it best- 'I don't want to live on this planet anymore.'
Edited to add- when I was young I caught wanderlust pretty hard and traveled most of the US, some of Mexico and Canada. I dreamt of visiting other countries and continents and was hugely disappointed to not find a way before having a family and obligations. All that went out the window about the time I started watching Locked Up Abroad. Brokedown Palace had nothing on those stories.
This mentality is far too common among modern humans. I've suffered these people at banks, bill pay stations, travel situations, etc. It's especially annoying when you're facing them with undeniable evidence that the machine contains mistaken data and they still respond with "I'm sorry, sir. There's nothing I can do. It says right here that you owe {$random-money}." Machines can be wrong, and even more commonly, the people who enter data into those machines can be wrong. But then you get these people who can clearly see that something is wrong, but they refuse to refute what the machine tells them? Why?
I swear, when the machines finally decide humans are obsolete, we'll happily line up and willingly march into the kill-box…
The recently uncovered UK post office accounting scandal goes to show that the computer is assumed correct by default!
It's a weird mixed feeling of annoyance and pity when you run into one of these folks.
The thing that resonated with me was the approach the officers used. These types of jobs attract people who like the power trip. You are considered guilty unless you can prove innocence. There was no probably cause, no need for a random search. I know people in LE and I can only imagine how these people would react to similar treatment if "flashing their badge" didn't get them a wave through. I think this is the thing that people don't fully get about the surveillance state and encryption back doors in addition to facial recognition. If there is ever a "hit" on any of these things, you are guilty. LE will stand on you to try to get you to crack because the tech can't be wrong. There are going to be so many innocent people in jail in the future dystopia.
If you are a U.S citizen, they legally can't deny you entry into your own country. Nor can they force you to answer any questions beyond the most basic ones that establish your citizenship and ID. They can intimidate you and make you feel you should divulge more, but this is just bluster, and you have rights that are well worth exercising. The ACLU confirms this by the way: https://www.aclu.org/know-your-rights/what-do-when-encounter....
The "excuse du jour". Always blame the computer and you will have a free pass to do anything as you want without being blamed for it. I had seen a similar pattern other times in airports.
I remember a story where after 9/11 and during the dirty bomb panic that ensued, a network of radiation detectors was deployed around New York. Authorities identified many dirty bombers. It seems Al Qaeda had recruited cancer patients for some reason, and they all congregated around radiation therapy centers for some nefarious purpose.
Those people shouldn't be fired for infringing upon this gentleman's civil rights, they should be carted off to a secure facility for the severely mentally impaired, for the good of society. I don't want any of them to be allowed to operate heavy machinery (including vehicles and elevators), matches or knives; the risk is too great.
If a vaccine kills 1 in a million people, that's still hundreds of people in US and thousands in entire world.
But now, see, how most people just don't understand the concept of very large/small numbers and can't interpret what it means exactly.
There are tests with very high false positive ratio. For example, pregnancy test.
Now, you don't do pregnancy test all the time. If you did it 5 times a day every day, basically at least once a week you will find out that you are pregnant and the test would be useful. And then you go to a real doctor to confirm.
The way to use test like that is to use it combined with some other knowledge/process/situation. In case of pregnancy test you only do pregnancy test when a) you show other early symptoms of pregnancy and b) you had sex with possibility of getting pregnant.
Facial recognition could be useful but as any high false positive test, looses its usefulness when deployed on large scale, indiscriminately.
I think the argument is that this particular form of innumeracy should no longer be acceptable if you're making decisions where this matters. At a minimum, this would be police officers, judges, lawmakers.
Unfortunately you can't build the system based on assumption that officers, judges or representatives are Vulcan and only use logic. Any system like that will necessarily be pretty bad.
The safety in the system is general public scrutiny, but as we see it is failing miserably, too.
Okay, then this kind of innumeracy is no longer acceptable in society, full-stop.
You know, this might be shocking, but around half of all people have less than average intelligence...
Do we euthanize half of population?
You know, then a significant portion of surviving population will have less than average intelligence...
Statistical literacy isn't so low we cannot make it a requirement to be a judge. The gatekeeping for once would be the start of a solution - make statistical literacy a qualification for well paying jobs and many will steer themselves to it.
The point is, for facial recognition to make sense, you would need to have some additional context.
For example it could make more sense if you followed up any results with an actual person making determination (some variation on a blind or double blind test).
Well of course anything mis-applied is probably bad. But sometimes false positives can be a good thing. For example, in immunology, it's quite common to use cheap tests that have quite high false positives, but low false negatives to screen samples before applying more expensive, but more rigorous tests.
> Drug tests generally produce false-positive results in 5% to 10% of cases and false negatives in 10% to 15% of cases, new research shows.
Lots of innocent people were probably jailed because of that. Afaik, most countries still relies on these tests to at least detain people. The system will implement something (anything?) as long as it is "good enough" or appear to fix the problem.
Most often, some tests have high sensitivity (no false negatives) whilst others have high specificity (no false positives). (Rarely both.)
So, for example, this is the reason why a positive RT-PCR test has more weight than a negative test because of the test’s high specificity but moderate sensitivity.
Lumping together false positive stats (for low-specificity tests) with false negative stats (for low-sensitivity tests) is misleading at best.
> Lots of innocent people were probably jailed because of that.
Efforts to increase the accuracy of the tests have downsides too, since the tests are already too sensitive. Lots of people have lost their jobs due to erroneous true positive results from eating something like a poppyseed bagel.
The test result is erroneous because no drugs were used, but a true positive because the trace amounts detected from the poppy seeds were actually present.
Improving the test can't fix this.
"Okay, let's run this through the system."
"What, no match? This system sucks."
"Let's try the competitor's system."
"56% match! We're in business."
I mean:
> Williams was wrongfully arrested in 2020 for federal larceny after he was misidentified by the Detroit Police Department’s facial recognition software after they used a grainy image from the surveillance footage. He was then picked from a photo lineup by the store security guard who wasn’t actually present for the incident. According to his testimony, Williams was detained for thirty hours and was not given any food or water.
So the software pointed out that some guy resembled a grainy photo. Okay.. that's like a random person calling in a tip that they saw someone who looks like a grainy photo on a Wanted-poster.
Then the guy gets arrested, there's a lineup for a security-guard who wasn't even there, they hold the guy for 30-hours without food or water, and that's due to the software?
The argument's too absurd for me to believe that it's being made in good-faith.
> The argument's too absurd for me to believe that it's being made in good-faith.
It's the US, so I have no difficulty believing this kind of thing can happen. All it needs is an overzealous DA office and state prosecutor with a hard-on for deploying these technologies to keep their conviction rates up (or think they can).
See also Aaron Schwartz, who thought things would get so dystopian for that young man, just for downloading a bunch of science papers?
All the facial-recognition software is supposed to do is point out similarities in pictures, and then humans can see if they agree. That's it. None of the silliness after that is due to the software, anymore than it's due to the security-camera that captured the grainy-footage in the first place.
This man was not "Wrongfully Arrested by Facial Recognition". He was wrongfully arrested (and criminally treated) by incompetent, capricious people and the bureaucracy that shields them. Bonus for the media outlets that could not care less about humanitarian rights violations or human dignity, but do have an axe to grind with Big Tech.
An accurate headline would be: Man Wrongfully Arrested by Corrupt State Government Tells Corrupt Federal Government His Story
It's disgusting and amoral and dehumanizing, and that's why I hated the job even though I had no direct involvement apart from running background checks and processing evidence.
That seems bad enough like someone should've tried to fix it by now. I guess there'd be issues making it difficult to improve?
And I'm not saying that my experience or the man in the article's experience is "normal", but it certainly does seem widespread. There have been efforts to improve the situation in pre-trial jails; the PREA (Prison Rape Elimination Act) has had a measurable positive effect on it even though its focus is more on post-conviction incarceration, but I have observed jail staff laughing at and making fun of county jail PREA training (I worked as a training officer for a while at the end of my career, my effort in trying to make things better). There is simply a lack of empathy and compassion that is part of the job requirement; if you show a heart you are pushed out of the career by the heartless ones. By and large, especially at the ground level in booking/processing/housing, the workforce is mostly bullies and sadists with a few more-or-less "normal" people in the mix to try and keep it under control.
It varies by jurisdiction, but in most cases you can be arrested on a warrant and held for "first appearance" before a judge, where you are either formally charged by the court or the charges are dropped. If you are arrested on a Friday evening in a jurisdiction that doesn't do weekend court (usually smaller cities and counties), you will stew in jail until said first appearance on Monday. If the charges are dropped at first appearance (which is usually held by videoconference) you are released but the arrest remains on your record until you have it expunged. If you are formally charged at first appearance, you then either receive a bail/bond amount, are released on your own recognizance, or else bound over for indictment. Once indicted, you again may be able to make bail unless the charges/circumstances determine you are to be held without bail (i.e. you are considered a danger to yourself/others/your alleged victim, or you are a flight risk).
- Dependence on snake-oil software
- Dependence on a known-unreliable but very entrenched technique (lineups)
- Inhumane treatment of prisoners.
Of these, the snake-oil software is probably easiest to go after, as it's newer. I assume he's made complaints about the other stuff, too.
I mean, I'd be shocked if it had happened here, but the latter two points, unfortunately, don't sound too out-there by the standards of US law enforcement.
They're not exclusive to the US, of course. But in the US, such detentions, or asphixiations, or shootings, or deprivations of property ("civil forfeiture") by law enforcement and/or under colour of law, are why the streets have been in rightful protest and upheaval for much of the past decade.
Those protests are a continuation of those of a half century ago and more, many based on the broken promises of the civil rights reforms of the 1960s.
And whilst the victims of such actions are overwhelmingly traditional minority populations, the abuses are not exclusive to them. Anyone poor or othered is particularly vulnerable. Even city councilmembers, state representatives, congressional representatives, and senators have been beaten, detained, or denied boarding privileges on flights due to Kafkaesque procedures and brutish enforcement.
It’s not fundamentally biased. Many implementations have had imbalanced training sets, which is a different conclusion. But you can’t say CNNs are fundamentally biased against women and people of color. That’s just plain wrong and makes the entire argument look poor.
And it isn't merely a question of unbalanced classes. The "bias" is, in part, arising from the inability of machine algorithms to apply rich models to datasets; and to treat "alike images" "alike" -- because the machine has no model of "alike".
In the case of images, luminosity values on an image sensor are not constitutive of human facial identity. Any algorithm unable to derive actual identity markers from these is necessarily fundamentally limited -- and routinely, these limitations lead to bias.
Actually they would. That's where the "all Asians look the same" comes from. Skewed training set.
Machines aren't fitting a morphological model. They are being fed 2D pixel patterns -- this information is profoundly insufficient.
"Contrast bias in pixel patterns" is not the same as "needing a few more cues to fit a morphological model". The former can never be made into an example of the latter; and therein is the problem.
Humans have rich models of other humans which we "fit" to the extremely shallow data present in a photograph.
This "fitting" is prior to our perception, so as soon as you're perceiving you're also using vast amounts of prior knowledge to reconstruct your environment.
The actual "photos" contain almost no information at all -- and lead to gross "stupid" errors.
The best CV models are often fine tuned from more generalized models that can recognize thousands and thousands of different kinds of objects, even if in the end they only need to find 1 kind. This is because they build generalized representations of the world.
None of this is to suggest there any bias that fundamental to them. We’re starting to see fruits of multi-task learning where models are integrating across domains, which ends up improving accuracy even for just one modality by incorporating more world knowledge. So it’s not like we’ve reached some kind of theoretical limit as to what they can do, and it turns out that theoretical limit is sexist and racist. Nothing could be further from the truth.
they are models parameterised by pixels, not by jawlines
to mistake one for the other is to mistake and misunderstand intelligence
we are not in a humean nightmare of sensory input -- we have bodies which have causal contact with reality, and from these bodies we have built a structural understanding of reality
our perception applies this structure to our "sensory input" -- thinking doesnt terminate in the visual cortext, that is where it begins
by the time we are seeing we are seeing vastly more than is present on our retina
What? People produce biased results ALL the time based on the 'datasets' they've seen.
Additionally, in this specific case humans looked at it, too and made the exact same mistake.
Even balanced training sets produce disparate levels of accuracy between races. It would seem computers are just less capable of recognizing certain faces.
I've never heard it also affects women though.
The tech provided a candidate, but all the decisions were made by humans, who came to the same conclusions.
I don't see how this is much different than artist's sketches or just using the video itself, other than cops might put more undue stock in the software, and the obvious potential for mass surveillance.
The fact that a media hype and a bunch of workers overestimating their AI "codes" have creeped in our daily lives is a simply appalling and shows how mediocre today's society is. This tech is not yet ready for what its being used for and should take a sidestep for a long, long time. Perhaps an aide for filtering large datasets but not much more.
This treatment is inhumane and borderline torture. Especially for a SUSPECT. Even if it had been the right guy, this is far from an acceptable way to treat fellow humans.