Apple sued by teen wrongly accused of shoplifting by unreliable facial-rec tech
theregister.com
theregister.com
They then put out a "BOLO" with the innocent man's name, had him arrested, figured out they have the wrong guy, do nothing about it and had him arrested again.
When Bah appeared in court to answer charges and Apple was asked to present the video that would have cleared Bah, Apple claimed that the video had been deleted. The video was later found by Bah's attorney during discovery. This happened more than once. A warrant was issued for Bah’s arrest and contained the photo of the imposter. Bah was arrested even though the he doesn't resemble the imposter, other than being Black. Prosecution against Bah continued in multiple states through June 2019.
Hmmm, seems like a lot of people involved in this case should be fired, including the Apple Employees, SIS Employees and Police officers, reguardless of whatever other outcomes there are.
I would like to see people who cause others’ freedom and time to be stolen from them because of laziness or malice to be charged with felonies.
Police1 - Yeah we saw the ID and stuff and some video
Innocent Man - where's the video, coz i was in different state
Accuser1 - we deleted it - poker face
Police - poker face
Innocent1 - poker face
Court - ok bye c u later
Examined and secured. Having evidence in official custody should be a prerequisite for using it in an official capacity - or stuff like this happens.
You don’t have a right to fight an arrest warrant. But you do have a right to review the evidence that will be used against you during your trial.
I think prosecutors should go to jail if they keep their evidence in a building covered in kerosene and then "whoops" the evidence has gone up in flames.
Blameless postmortem are fine only if steps are taken to uncover and fix the problem at the root cause. There's no excuse for systematically jailing innocent or pushing buggy software.
In this case, what steps were taken? Video was deleted!
Does that apply to developers of an airplane's autopilot, insulin pumps, and other stuff that gets people killed?
On the other hand, if someone is sabotaging the system for personal gain, they should certainly be imprisoned.
How might a government systematically disincentivize this sort of malicious and negligent behavior without it backfiring? And why hasn't it happened already?
I imagine the matter has been thoroughly explored by legal scholars, but I'm having trouble finding publications. Can someone point me in the right direction?
I'm in a safe environment where i can say i made a mistake and document it. And still i have a hard time doing it because how i grew up. It was much safer to be quiet or lie and get away than trusting authorities or parents to not punish me.
But removing that threat would change how the system works and how ppl deal with mistakes.
I basically learned that because I hated the bosses that would take credit for their subordinates ideas and then blame them when something went wrong, so I'm trying to do the opposite.
"never admit to the system that you F-ed up" is a lesson every kid should learn by the end of grade school.
Let that run for a generation and stuff like in TFA is what you get.
this essentially exists in the united states with the dual state and federal systems, but the local systems are extremely resilient against investigations, and the federal system has no interest in actually performing this role.
in rare cases there have been FBI investigations and federal oversight of entire departments, such as the Seattle consent decree, but there are many problems with such arrangements as covered by local journalists and activists. i'll see if i can dredge up some links later.
It also lacks police oversight; the UK has https://www.justiceinspectorates.gov.uk/hmicfrs/ and https://www.policeconduct.gov.uk/ , which while they are far from perfect are at least organizations with the capacity to put out a report saying "here's what went wrong and how it should be fixed".
In the US all police accountability is local, which means that the >51% of the voters who prefer being tough on crime (more false positives in the hope of more "true positives") get what they want.
aclu has a timeline https://www.aclu-wa.org/pages/timeline-seattle-police-accoun...
federal oversight means the consent decree is subject to national politics that may be alternately undermining (executive branch republicans promote impunity) or disinterested (executive branch democrats fear criticism of reform efforts).
the decree is also affected by national events and attention. the late nationally-motivated police reform efforts are running into roadblocks as officials claim those goals conflict with compliance.
https://crosscut.com/2020/06/debate-over-policing-seattle-co...
https://crosscut.com/politics/2021/05/where-seattle-police-r...
notably, an internal police watchdog body created by the consent decree and charged with investigating officers, may only recommend action to the chief, who can simply reject it. the watchdog body is staffed by police and not civilians. there are some examples in the timeline above but here's one from last year.
https://www.seattletimes.com/seattle-news/politics/seattle-p...
and the police union is effectively able to write law that undermines oversight. the union contract actually supersedes city law on police accountability. this is a failure of the city council but was agreed upon by the overseeing judge.
https://www.thestranger.com/slog/2018/10/17/34045061/cpc-thr...
after nearly a decade of effectively resisting federal oversight and reform efforts, spd has developed an attitude of total impunity. a protest marched to the SPOG office last year, and at no provocation was dispersed with teargas and impact munitions as officers blasted a pop country song over loudspeakers. https://www.thestranger.com/slog/2020/09/08/44432799/police-...
Done
This changes the whole game not in favor of Apple. And adds few zeros to the cost for Apple to try to wiggle out of this clean and not admit racist bias.
Welcome to america.
The white guy would get $10 off apple store offer from apple if he was lucky.
But not in this case.
I’m totally on the side of the guy.
How is this even possible? Why don't judges recognize this obvious abuse and stop it?
It’s by far not on an extreme side
I disagree that it is just Americans. In real terms language evolves on its own and people tend to imitate the pieces they find useful for one reason or another. I remember living in the old country as the Iron Curtain fell. All of a sudden, adding English words into just about any conversation was a thing to do. Some of it was, because word equivalents were not quite ready yet; some of it was fashion.
I think cop porn is just another language phase in US.
Perhaps, or a culture should be created where there should be no fear of negative repercussions for taking ownership of something? Seems like a typical organizational thing where responsibilities are not clear and people are afraid to take it.
Edit, not saying this is not unforgivable. Just saying a critical look at the organizational level would be good.
Firing these people is both just and pragmatic.
The story is still horrific, but I am mildly miffed.
"Fired"? This is tampering with evidence, contempt of court, filing false reports and so on, all done repeatedly. People need to go to jail, because people go to jail for those crimes and for lesser crimes as well.
If Ousmane Bah didnt have a good lawyer he would be in jail and have his life destroyed already, such actions like this should be treated as a crime of falsifying evidence at least.
There must be many more that are not lucky.
Around 4.1% of people on death row are estimated to be innocent.
https://www.pnas.org/content/111/20/7230
I have a hard time imaging, that the rate is lower for lesser crimes where the victims of the justice system can be coerced to accept a "lesser" punishment.
Personally I believe the 4.1% estimate to be pretty low, and it one of the reasons I strongly oppose capital punishment, I simply do not trust our legal system enough to give them the power and authority to kill people
Agreed, because humans are fallible, and a jury is made of humans The courts can always release on appeal. You can't un-kill someone
That said in broad strokes, Death is final, and while reintegration may be pretty hard, if a person spent 20 years in prison because society made a mistake, I think that person should get a very large settlement from the state to the point where they never have to work again thus making that transition much easier.
I am also a supporter of legalized suicide for the general public as well, especially for the terminally ill
Combined with some therapy or something.
But not as punishment.
For me, the main reason to oppose the death penalty is also that innocent people get killed.
Remember kids, the criminal justice system has a strong bias towards conviction if charges are filed, so don't talk to the police. You're only helping them build a nonsense case against you, even if they've got another suspect you never know when a rookie cop is going to decide that getting you on Jay walking is good for his numbers this month. Unless you personally witnessed a serious crime like assault, rape or murder, shut the fuck up.
And even then, only share your observations of the crime in question. If the conversation turns towards you in any way, shut the fuck up.
And never forget the only things you should ever say during a traffic stop:
Why have you pulled me over?
I don't know. Here are my license and registration.
I'm not required to discuss my day.
When was your radar last calibrated? (avoid this one unless you really want to piss them off)
Am I free to go?
Am I under arrest?
I would like to speak to a lawyer.
If it's not on this list, have your lawyer present and consult them directly before saying it.
Can someone explain the logic of this? What is the purpose of an ID that shouldn't be used for identification? And why would anyone in their right mind issue a non-biometric ID in the first place?
Most places you won't see a temporary permit like this as they'll have a machine on hand to make the license, but I suppose not every location has that available. It is only meant to last until your photo and information gets to wherever they print the cards and your card gets mailed to you (less than 2 weeks).
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ab...
Not that it really matters/makes a difference, but presumably it's 'LookOut' - at least, that seems less weird/forced into a specific desired acronym.
As a noun 'lookout' (or 'look-out') is from verb phrase 'look out' anyway.
Why would anyone routinely delete evidence of crime instead of making a copy of it before it was deleted? What's the point of having surveillance equipment and footage if it's just routinely deleted even if there was a theft?
>The video from an October 2018 theft misattributed to Bah in Rockaway, New Jersey, was also deleted.
Oh wow, again?
>And as it turned out, the video of the Boston incident turned up eventually – Bah's attorneys found it during the discovery process. It showed the impostor, not Bah.
They should open a criminal investigation into what happened with the video. Intentionally accusing someone they know to be innocent, taking their time and freedom is completely inexcusable.
(speculation) because the prosecutors preferred to ruin an innocent man's life over appearing incompetent... it technically risks evidence tampering, but they can probably just argue it back to incompetence so they have nothing to lose.
So essentially a kid stole another kids ID. Because the (thieving) kid had gotten in trouble with the same ID before it was also a match in the database.
It does seem like a true failure but the headline makes it sound like this was an AI falsely accusing someone based on pure randomness. Even without the tech, the police would have had the name of the wrongly accused person because the security guard only had the fake/wrong ID and the person's ability to match faces was not great.
> "The arresting officer was able to identify the impostor as Mamadou Barrie, a friend of the Plaintiff, who apparently stole the learner’s permit from the Plaintiff," the complaint says. "These arrests specifically [noted] that Barrie had pretended to be Ousmane Bah."
> The detective subsequently submitted an information request "to the NYPD’s Facial Identification Section (FIS), which identified the photograph as potentially depicting two people, one of whom was purportedly Ousmane Bah – and the other was the actual thief, Mamadou Barrie."
no, it doesn't. the headline makes it sound like facial recognition is unreliable for some reason.
the reason, in this case, is that the Apple store and/or SIS personnel gave the recognition system Ousmane's name when they weren't able to prove it was him, and the document provided for identification was not actual proof of identification.
Further facial recognition matches assigned further blame to the misidentified person, increasing the evidence against him.
Then, because store owners and security firms really want to put away shoplifters, they claimed that video proof of the shoplifting was routinely deleted when there was no policy describing a deletion policy within Apple or SIS, only for video proof that the individual was not Mr. Bah to later be found.
in this case, facial recognition tech was unreliable entirely because of the people running it.
I sort of skipped through the actual headline trying to figure out how AI was involved. It was. But only after several blunders by people.
Drum up false charges and they get automagically mapped to the face with nobody able to undo anything. They're now forever a criminal. Honestly, it sounds extremely dangerous in a field such as criminal investigations.
So instead we have scattered silos of partial and probably incorrect data in various places. FBI has a bit, Police department in state X has one, State Y's DMV has something wrong because X years ago this person was a victim of identity fraud, etc etc. It's a giant mess and no wonder the police and various entities in this example failed utterly. They're ultimately all victims of a system that has no single source of truth, yet we're tasking them with using "tech" to solve crime. So instead of fixing the source of truth, we blame the tech.
There is an incorrect tendency to ascribe intelligence to AI, which can make especially laypeople prone to assume that if an AI system makes a statement, that statement has a certain sort of validity to it that isn't necessarily justified.
Garbage in garbage out applies to AI just like it applies to any other software system, and society needs to develop more resilience to this fact.
The same would have happened if a human recognized the kid and matched him to the (stolen) id card on file.
Increasing the distance between database input and output gets in the way of human heuristics for recognizing and fixing such mistakes. Yes, the same could have happened without AI, but the probabilities are different.
In this case, it was obviously a mismatch between how the different operators treated the 'name' field - one operator put in an uncertain ID, and another one trusted the system to be correct.
These stories are IMPORTANT. They might help someone think twice before either using or blindly trusting these systems.
You wouldn't title a story about a guy hitting his thumb with a hammer "Man attacked by unreliable hammer"
Wrong.
In this case the facial recognition tech itself was arguably not unreliable. The human operators were the unreliable factor.
The recognition tech reliably tagged the impersonator as Ousmane exactly as it was instructed to do. The system worked exactly as intended. It is the human operator whose intention was wrong.
This has nothing to do with AI being unreliable and everything to do with the employees of this SIS company going "yep kid's black, he's the one who did it" without half a thought.
This has everything to do with AI being unreliable. AI is unreliable. It was unreliable yet again here. Anyone relying on AI as the evidence in a prosecution needs to have their faces rubbed in the ignorance. AI is as unreliable as any human being. In fact, it is more unreliable because people understand the limits of the reliability of humans and look for corroboration and examine the evidence with a mind on its potential shortcomings. Yet there is still this pervasive and pernicious belief that "Computers don't lie" which is completely and utterly false and worth repeating often, with emphasis until it is part of human collective understanding of the world to the same degree that water is wet.
"Computer says x" by itself counts for nothing. Anyone presenting evidence like that by alone should be presenting a huge red flag that something nasty is going on, via either malice or sheer incompetence.
The problem was that one human recorded a name that they should have known was not real, and another person read that name in a report and arrested someone with that name. The story would have been no different if a human had correctly recognized the same person in both videos.
Sure you can argue that the pure tech worked as designed, but the system includes the human element as much as the tech element.
These "operators" that you say are part of the production execution of "AI" are also part of every single institution that led up to this person being here and existing. Everything from registering their birth, capturing their DMV details, capturing their details when enrolling at a school and opening a bank account, taking their photo for the driver's license, etc. All gooey, imperfect, corruptible and fallible people that taint the "chain of custody" about the person's identity.
Separating the two things (operator and technology) is merely a technicality, in reality they are not transparent to the general public and should be treated as parts of the same.
If AI™ only happens in a black box behind closed doors with people getting the opportunity to make inconvenient results disappear and that is basically the way it is trying to be established then no, it is not reliable.
The operator is part of this whole system, if it can't be used without the unreliable operator, the tech is not reliable.
You could train a fake money detection AI with cat videos from youtube, it would probably do something with images of cats. However I hope no one would try to argue in front of a court that the resulting AI would be reliable at detecting fake money. In this case a stolen "do not use as id" slip was apparently good enough to serve as input validation, I would be surprised if their database wasn't overflowing with bad data.
The AI correctly identified that the faces of people in two videos were the same person. It did it's job. The person responsible for correctly identifying the person in both these videos failed to do their job correctly.
Maybe other people didn't read the headline as I did. But if I hadn't bothered to click on it I 100% was just going to assume it was because the tech itself was unreliable and they picked out someone who looked like him.
Perhaps if I wasn't read up on the debate around this I wouldn't have thought that - but I imagine most people would assume this with confidence if they saw the headline on their feed.
Note that we are both talking about other people - we, of course, are not like that!
There are real people who suffer real consequences because of the programs we write. When things go wrong we don’t get to point the finger at the algorithm and the users and take no blame as the developers.
1. the plaintiff argues that in Boston case (page 6) apple store only took less than 10 minutes to identify the shoplifter, so they probably used facial recognition tech. Note the shoplifter left the store, apple found the incident, report it to police within 10 minutes, without actually detain the person.
2. I'm guessing the facial recognition is actually accurate because it was based on their previous security video, not plaintiff's real id photo(which apple probably don't have access all the time). So the facial recognition just linked one person with previous security video properly, and the id of person was based on previous record made by employee.
3. Thus there is nothing wrong here with facial recognition, but the title will definitely evoke fear and people's emotion since most people will not see the details.
"Where we're you that evening? do you own a outfit matching the suspect? Etc." This kid would have been quickly cleared. It's already terrifying to be questioned by the police when you are innocent. How do you think he felt when they said they had video evidence of him shoplifting. They already made up their mind.
Now replace your colleagues with an AI. Simply because you have a match does not mean you have the suspect. Just over a year ago we were discussing the same issue: https://news.ycombinator.com/item?id=23628394
That depends on if the goal of those using the technology is arriving at the truth or if the goal is naming a suspect who you can reliably stick a guilty verdict on.
They had a system of levels of confidence they would use in their reports when identifying someone.
The highest level of confidence was achieved when a close friend or family member identified the person in the picture of video.
There were progressively lesser levels of confidence for people less close to the purported subject of the image.
They also had a system of points of similarity/difference. but they normally only used that to establish that the same subject had been observed by multiple cameras, rather than to establish the identity of the subject.
You overestimate the initiative and competence of the police.
Apple and SIS have a qualified law enforcement privilege that allows them to err in store security-related accusations and not be sued for it. However, if they exhibit "reckless disregard for the truth" – ignoring obvious facts, for example, they lose that privilege.
If it's the latter, then Apple is in serious trouble and they'll probably end up paying this kid big time in a sealed settlement so they don't lose this privilege.
Would love to know.
The name was picked up from a lost learner's drivers permit, that was found on an suspect while in custody, the permit did not have a photo and then this stolen name had been propagated further and attached to other cases. They used a paper without a photo as identification. It actually says right on it, that it is not to be used as identification, WTF?
> In addition, it asserts Bah's apprehension was in part due to the application of unreliable facial-recognition technology in the shoplifting incidents in New York.
Vague statement, unclear what exactly role of the facial recognition was.
> The detective subsequently submitted an information request "to the NYPD’s Facial Identification Section (FIS), which identified the photograph as potentially depicting two people, one of whom was purportedly Ousmane Bah – and the other was the actual thief, Mamadou Barrie."
Seems like the software did what it had to: identified two people. The article also says that they were friends, so it is not surprising that were both present on some frames. It can be source of some confusion as well, not tech related though.
> The video from an October 2018 theft misattributed to Bah in Rockaway, New Jersey, was also deleted. Apple and SIS are said to have told the New York court that neither firm has any written policy on video retention.
Yeah, this is fishy. I am very doubtful they don't have any written retention policy.
Human error, eh?
However, on that note, does anyone have a photo of such permit? All Google images of 'learners permit new york' have photo IDs; the closest thing I can find is the post-test temporary full license[1], which doesn't include information like the person's height. The exhibit is also restricted on Pacer[2], likely because it contains PII.
0: https://regmedia.co.uk/2021/05/29/pacer_bah_apple.pdf#page=3
1: http://2.bp.blogspot.com/-c7SAttW166s/VaOtXtrptTI/AAAAAAAABc... (linked from this public blog http://invisiblevisibleman.blogspot.com/2015/07/a-driving-te... )
I have a temporary non-learners license but due to PII I'm not taking a photo of it. It is a printout that contains everything except a photo (so name, height, eye color, etc.). It's used as a temporary license until the real one is mailed to your address. This is distinct from the post-test printout which would be, I'm guessing, used in conjunction with a learners permit license. The temporary license printout is used on its own without another permit which is why it contains more info.
How do I also get qualified immunity
Or is it dependent on how much money and influence you have? So money can buy qualified immunity that's pretty sweet deal.
Why would there not be a written video retention policy?
that is the gist. They don't care because they can do all that stuff and can't be held responsible.
https://www.mercurynews.com/2021/06/01/santa-clara-county-ju...
Now, as then, facial recognition is being used as clickbait, when all that really happened is that this guy stole an ID, got caught shoplifting in an Apple Store with that ID, and thus had the information on the ID incorrectly associated with his face—either in a facial recognition database, or simply by the photo being used with this stolen ID.
My understanding from the previous story was that it was not Apple that took the information from the ID, but rather the police, who gave that information to Apple. It looks from this story like there's also a third party involved, SIS, the security contractor, which, of course, further muddies the waters.
The developer who developed the system to falsify EPA results for Volkswagen went to jail for 3 years. The judge specifically cited that he had a professional duty to not commit illegal behavior but chose not to because he didn't want to lose his job.
Ref: https://www.nytimes.com/2017/08/25/business/volkswagen-engin...
Based on how key software is in so many things, we're going to be faced with a choice soon:
a) As an industry, clean house and hold ourselves accountable to better practices, uncovering and fixing honest mistakes, and punishing misbehavior; OR
b) Do nothing and let regulators write the rules for us.. which we know will be outsources to the big consultancies like Deloitte, BAH, PwC, KPMG, etc.
If we don't all get onboard for A, the alternative will be ugly.
Can someone smarter than me explain what this line is referring to? Sounds scary.
What the heck are "iStores"?
https://9to5mac.com/wp-content/uploads/sites/6/2013/03/istor...
This whole sub-contracting policing out to private companies is going to end up being a constant state of witch trials. At least the state is the law. These two companies aren't in the slightest.
Because there's extra requirements for voting that aren't related to actually having a government-issued ID card.
But the point remains: if an ID card isn't considered valid for voting, then why should it be considered valid for identifying an individual for a crime? It is a valid ID (identification record) or it is not.
We consider the presentation of a credit card to be sufficient to identify you for the purchase of a product.
We do not consider the presentation of a credit card to be valid to identify you for a crime without corroborating evidence.
> This temporary document is not to be used for identification purposes.
The thing is that it didn't - he's getting heckled due to warrants for his name, but charges against him have always been dropped when the DA has seen 'oh this isn't him' - not because these DAs are nice people who see the misunderstanding, but because the evidence is not strong enough to prove he committed the crime. That's why the case is for "320 Assault Libel & Slander" against Apple & the security company, not false imprisonment against the police department(s) (who have reasonable cause to detain suspects).
If you showed up to vote with a learners permit, you would get some funny looks and then be shown to the voting machine. Nobody asks you for ID at all. That's what the current political controversy is about – whether to even require any ID.
It's also a total non-sequitur from this thread.
I wonder if there are such datapoints of clearly wrongfully accused white people where they still had a BOLO put on them despite no evidence.
They fired employees for "lesser" offenses.
This attitude is why the world doesn't like the US. You are not the center of the world, many countries fare much better than you do.
Furthermore, on a global scale, the trends of increased energy demand, increased population growth, increased carbon emissions, increased ecosystem degradation, to name just a few of the main actors, are overwhelmingly positive, and hence these are global issues that affect the sustainability of the entire global society. It doesn't matter if Luxembourg or Sweden develops a socdem utilitarian net-zero utopia, they are just a blip on the radar. All of the third-world countries that the first world exploited over the past centuries in order to fuel their own growth & industrialism, are now all fervently contributing to the above trends, to the extent that all of the first world's recent attempts to dial back their consumption, emissions & so on are laughably futile.
The only possible way to change these trajectories would be for the majority of capitalists across the world to all drastically alter their priorities and how they operate, but this is absolutely not plausible, given that no corporation has ever not sought to increase its profit and growth every quarter; if they somehow hypothetically did, the system would come tumbling down and simply cause collapse even sooner.
They all have a good standard of living, healthcare, not much crime, and the state helps you with social services if you mess up. This is quite different from the late-stage capitalism that is currently going on in the US...
This isn't a case of a facial recognition software failing to distinguish two black faces or people going after a black man disproportionately, this is a case of a repeated shoplifter giving a fake name, and the person whose name was given being arrested.
There are real cases of systemic discrimination and racial bias having terrible effects on peoples lives; crying wolf makes it harder to get people to recognize and address the actual problems.
This is an absolutely classic example.
Also this person was not arrested repeatedly. He was arrested once and then had the charges dropped when it was clear he wasn't the right person, and then received a mailed notice of a warrant for his arrest from a different county two weeks later for a separate offense (both instances of shoplifting falsely attributed to him). In a third precinct, after a third incident, a different person with the same name was summoned, and also quickly identified as not being the right person.
This is a case of the left hand not knowing what the right is doing. It is a problem that can have serious consequences, but it is not related to racial bias.
This pretty much sums up modern days Apple. In terms of above the law, being righteous. And it wasn't a honest mistake or one-off incidents. They have been doing it repeatedly. It just reeks of hypocrisy.