This is also comparable to self-driving car: Waymo's system has achieve the level of "less error than human on average" long time ago, but they are still only doing slow expansion. Why? Because when AI is involved, human naturally expect them to be almost 100% correct, not just barely better than human, otherwise why bother use AI (and all the troubles to legalize for AI) in the first place? So in this case, I guess the police just blindly trust the AI being 99.99% correct.
It’s a camera system, and data can be queried by police. They have to search for something. So the cops searched for a vehicle matching the description in the general area, seemed to take the first one, and called it a day.
That’s not an AI-uncanny-valley. That’s the equivalent of (pre-flock/cameras) looking at vehicle registration list and assuming the one at the top of the list must be it.
The automation-trust is a legitimate concern but it’s not applicable here.
Flock does have AI components (this one seems LLM based). Basically every ANPR system will also use some level of AI.
Also, on "reality would like a word". Why not just speak normally instead of using what seems like a high-school grade comeback?
The main Flock related issue here is that officers treat results that come out of the systems as gospel. The same thing has happened with face recognition systems. There is also the broader issue of access controls to Flock, 5A, and whether places should be building up a technology so powerful (surveillance infrastructure can easily be used for evil).
And the issue is absolutely not just cops taking what they say as gospel; that’s a teeny tiny itty bitty fraction of the issue, which is in itself reductive. This is a massively multifaceted issue and to state otherwise means you don’t understand the scope.
Perhaps you should actually read TFA. Here’s another one. https://cybernews.com/privacy/flock-ai-os-investigate/
So saying you said something because my non-existent position is "absurd" and "clearly false" indicates massive levels of confusion.
You then go on to imply I haven't read the article, using an acronymised swear (I'm not a pious person but clearly directing a swear to someone makes a message come off as aggressive), even though your understanding of what I said is based off non-existent messages. (This is quite ironic).
Flock is a company. Companies are a type of organisation: it makes no sense to describe a company as AI, as AI is a type of computer system. It may make sense to say Flock is an AI-based company, but calling them "AI" in itself makes no sense.
I can't find anything indicating that an LLM was used in this case, and this county seems to mainly have Falcon cameras (https://www.scribd.com/document/983922612/250730110407Flock-...). Reverse engineering seems to indicate these are powered by non-LLM AIs like Tensorflow vision NNs. These have been around for ages, and notably no-one is alleging these models hallucinated etc.
People also (perhaps wrongly) use "Flock" to refer to the camera system. CrendKing explicitly calls "Flock" a "camera system". A system of cameras is not AI, and the point of the comment seems to be that it's inappropriate to bring up LLM related AI fears about this system.
Perhaps more facts will come out, but if we take the article at face value, the cops are at this woman's house, looking at her car and can see with their own eyes that the car right in front of them has obviously not been in an accident. All that is required is for these cops to rub two brain cells together and they will reach the conclusion that Flock misread a license plate.
It's also worth emphasizing that this wasn't for a parking ticket. The extremely low level of care with which these public servants are conducting a manslaughter investigation is somewhat alarming.
Yes, but it is still your fault. “I am dumb and I was deceived” is no valid defense, least of all for a police officer.
Gross lack of due diligence: it has full fault.
Untrained officials: graver systemic fault.
> company, are totally innocent
How clear is the uncertainty of the output?
It is not even clear from the article that there were anything wrong with the output. She has a car of that make. And it doesnt appear to be in dispute that she drove nearby.
It simply is the case that “being nearby” in itself is not enough evidence that you were involved in a crash. Especially when the colour doesn’t match and the vehicle is not damaged.
For the sake of argument, let’s suppose that Flock is 100% evil/guilty/blameworthy in this case. Even in that case, two things are still true:
A) Flock has no authority to arrest anyone B) Flock has no authority over the police
The buck has to stop somewhere. It’s counterproductive to focus on what Flock did, compared to what the police did. And I say that as someone who is not a fan of Flock.
Police approving AI PRs without reading them: This is the way the world works now. Get with the times.
It’s more like normal people don’t know that 97.8% accurate isn’t 100% - despite how good of a score that would be on a test.
Law enforcement is predominantly made up of normal people.
You might see where this is going.
The number one demograph for domestic violence and abuse are cops, they are not just a randomly selected group of people that chanced into law enforcement as a career.
Be sure you don’t have the causation backwards - or that it doesn’t flow strongly both ways!
It attracts certain people - and being around that crowd and power pushes some number of people over the line that were near it.
See also: stimulant use, abuse, in the construction industry.
Closer analogy: The dumbest student in a class tells you 2 + 2 = 5 and you believe them.
Even more apt analogy: You're the dumbest student in the class, the second-dumbest student tells you 2 + 2 = 5, but you lack the critical thinking skills necessary to know whether that student is smart or not so you just believe them.
Yes, the system is broken even before we get to Flock (or whatever tech) but Flock makes it worse. If a bad tool is badly used to cause harm, it’s fair to criticise its existence. You can’t fix police abuse with a snap of the finger, so you need to start but taking away the tools which facilitate it.
If it were your ass which had been thrown in solitary, would you be pleading with your lawyer to “go easy on Flock. Sure, they allowed a major disruption and trauma into my life, but they’re just a tool”?
If a kid has shown themselves to be repeatedly irresponsible, when they ask for a circular saw you don't just give it to them and turn away. You make them explain what it is for and ensure they know how to safely use it. And if they start swinging it around wildly in opposition to what you said, you take it away, they are obviously incapable of using it right.
If someone released a hammer that when held a certain way would cause your hand to explode and it kept happening over and over, the solution wouldn’t be just to tell people to hold it better, it would also be to recall the tool and fix it so the issue were less likely or impossible to occur. And if it can’t be done, then maybe the hammer shouldn’t be released again.
This “it’s just a tool, the proper solution is to rework civilisation” defence is frankly as tired as it is ignorant. The way you fix the system as a whole is via a lot of small fixes.
I don't it's about an innocent skill issue like you imply (holding a hammer the wrong way). It's an ethical issue of bad actors in the government, together with an accountability issue of the system preventing holding those bad actors to account.
AI cameras are just one tool; there are other tools, too. Where do you draw the line for which tools deserve the removal for your kind of reason? Perhaps degun the police departments because guns also get abused? How do you decide in a principled way that AI cameras are somehow special compared to other tools?
It doesn’t have to, but historically we can observe that the longer something has been entrenched as the norm, the harder it is to change (authoritarians regimes take advantage of this too, by making it appear some things are older than they are). Which is why it’s doubly important to resist when something new with bad consequences is introduced.
> I don't it's about an innocent skill issue like you imply
There was no “skill issue” implication. If anything, the opposite is the point.
> AI cameras are just one tool; there are other tools, too.
This is just whataboutism. Yes, there are several tools which are problematic. The submission is about this one, which is why we’re discussing it. We can’t discuss everything that needs to change in every conversation. That’s called focus.
The people who say there are problems with the human police officers are perceived as excusing Flock.
Who is responsible for the area inside a square, the width or the height? It is not an additive relationship.
And if your country happens to have a police force that is historically bad at its job, giving them that power seems like a really stupid idea.
Certain technology just has too much potential to be misused / people repeatedly prove they can't use it properly. It also comes with risks of making authoritarian takeover easier.
However, here, it seems they used Flock a bit like a regular ANPR. The actions they took independent of Flock show there are better stories to use to criticise this technology.
A lot of people coming to this story seem to have the idea that it's unthinkable for someone to spend 13 days in jail when they have clear evidence they're innocent, so there must be some novel explanation for how this could happen. But as any criminal defense lawyer will tell you it's very thinkable. She was ordered to report to jail on a Friday afternoon, the standard stratagem cops use to ensure people they don't like spend at least a weekend behind bars. She was denied bail, even though she made no move to flee in the months where she knew the police thought she did it, because the courts in many places view mandatory pretrial detention as a punishment for presumptively guilty people.
If a random person walks into a police station and accuses a random stranger of murder, and the police put that stranger in jail based on that alone, the main problem is not the accuser.
A scientist who uses an LLM to write a paper remains responsible for its contents. And authorities that outsource police work remain responsible for what comes from that.
This is the good old police incompetence that you come across once in a while and shake your head.
Where detectives lock on one angle of a case and completely refuse to entertain any other possible explanations
I have never released a single product that has seen people die through common user misuse.
The standards for solutions shouldn't get _lower_ as the stakes get _higher_.
If a private company can puts hundreds of tons of payload into orbit without ever causing someone to accidentally be imprisoned a company that sells cameras on a stick shouldn't either.