Hacker News Reader

Comment by ajb | Hacker News Reader

Seems like an opportunity for the lawyers to me.

Being evicted from a store is necessarily a public act. Telling someone wrongly that they are a thief, in public, seems like libel. Since most people go to stores in their neighbourhood, this could easily happen in front of your friends, neighbours, parents of other kids at your school, fellow members of social clubs or worship clubs, etc.

If this happened to me, I'd be going and seeing if I could get any lawyers to take this on a no-win no fee basis.

It seems like businesses are increasingly failing to think through what their process will look like to real human beings.

View on HN

When I worked on analysis software for police & intelligence agencies, it quickly became clear that no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible.

If people are going to treat these things like they are forensic evidence and not lead generation, then they shouldn't be legal. Some precedent that people need to check these things before acting on them would be very good.

From The flaws of policies requiring human oversight of government algorithms https://www.sciencedirect.com/science/article/pii/S026736492... :

    human oversight policies provide a false sense of security in
    adopting algorithms and enable vendors and agencies to shirk
    accountability for algorithmic harms.

And the better The Algorithm is, the less people check, and the more of its faults pass through.

Having worked on systems that required human oversight, I've seen first-hand, and analyzed data on, how people click the "LGTM" button without reflecting. We even noticed how when we improved our predictor with expert input, so that it performed better according to experts who worked with it, it would perform worse when we benchmarked on the previously "manually checked" data set. Analysis showed that the manual checks had let through so many of the algorithm's faults, that when the algorithm improved, it got a worse score.

    We present seven flaws of predictive optimization. Our aim is to outline a set of
    objections inherent to predictive optimization that cannot be easily fixed using a
    design or engineering change. Taken together, these critical flaws undermine the
    legitimacy of applications of predictive optimization.

    1. Good predictions may not lead to good decisions
    2. It's hard to measure what we truly care about
    3. The training data rarely matches the deployment setting
    4. Social outcomes aren’t accurately predictable, with or without machine learning
    5. Disparate performance between groups can’t be fixed by algorithmic interventions
    6. Providing adequate contestability undercuts putative efficiency benefits
    7. Predictive optimization doesn't account for strategic behavior

True, "check" was poor phrasing, because these things don't produce evidence, they produce alerts to guide investigation.

It should be a given that these systems don't produce anything actionable, they are a guide for a user to look more closely, of course, that requires someone in the chain stops and says "this isn't usable evidence".

Very interesting - thanks for giving us the benefit of your experience.

In the case of store workers, it's not obvious what other checking they could do. The only reasonable use I can see is for them to keep an eye on the person that has been flagged. But I expect that the entire purpose of this tech is to reduce the number of store guards they need to pay, so they are likely not to have the manpower. If the policy is 'don't let them in the store', I don't see how a store worker can avoid making false accusations.

There's something fundamentally wrong with these probabilistic analysis tools being sold currently as "artificial intelligence". The nuance in "probabilistic" is quickly lost and the output is taken as deterministic.

Any piece of software that puts out data that is acted upon should have clear disclaimers attached. Calling it "intelligent" is irresponsible.

"Calling it "intelligent" is irresponsible." How so ?

If a human of animal behaving the same were to be describe as intelligent, than so are those systems.

How so? People take it at face value and harm other people based on its outputs, as per the article linked.

> If a human of animal behaving the same were to be describe as intelligent, than so are those systems.

I cannot parse this sentence. Behaving the same as what? What bearing do claims of human or animal intelligence have on me stating that generating liability by trusting probabilistic analysis, and actively harming people, by calling software "intelligent", is not responsible behavior?

The bearing is on the claim that calling software "intelligent" have anything to do with trusting or liability. Dumb or not, no one is supposed to blindly trust it.

we can call it frediness, but the problem isn't what it's called, the problem is that the profit motive only works when there's competition. if there were several shops nearby and one of them started rejecting 80% of customers based on frediness, they'd go out of business and the others who prefer having a human guard would prosper. if it's the only shop within a 10 mile radius, and it only flags out 1% of innocent people as shoplifters, then the profit motive doesn't provide the balancing force. that's what regulation is for.

I can think of one possible regulation that I think would produce the proper behavior incentives.

Require that the system intentionally produce false positives (selected at random) at approximately the same rate as the true positives. This means that the users of the system will be aware that any indication has only a 50% chance of being "real", and therefore can be used as an indicator to increase caution but NOT on it's own as a signal of guilt.

Why not just flip a coin then? It would be much cheaper?

I imagine the solution space is much bigger.

I think it could be an actually pretty good solution for this. A bit like Chaos Monkey but for human systems. Introducing error often enough that operators don't assume it never happens.

If the expected rate of known shoplifters is 1 in 10,000 then the human security team can keep a close eye on 10,000 people (one of whom is probably a shoplifter) or on just 2 people (one of whom is probably a shoplifter). The latter is much more efficient.

Without the intentional false positives they could simply throw out 1 individual with no need to keep an eye on anyone at all. That's enen MORE efficient, but it utterly ruins the lives of anyone falsely identified.

they'll just throw both out.

Similar to story points and task estimates suddenly becoming deadlines

>If people are going to treat these things like they are forensic evidence

Are most kinds of forensic evidence actually any better?

Well, plenty of bunk stuff (bite mark analysis, hair analysis, etc...) has been used in the past, but in theory anything used today should be backed up by significant evidence and rigorous methodology, and obviously it'll vary based on jurisdiction, but in theory courts should require that background.

Certainly, more transparency than "we bought some facial recognition software, no, you can't see the code, that's a trade secret".

> no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible.

That's precisely my anecdotal experience.

Subtlety and nuance are easily ignored where their absence appears to provide easy answers and the avoidance of responsibility for their outcomes.

Similarly to "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It".

I don't trust a sales person or marketer to not hype such a system as being flawless whodonit machine. I certainly don't trust that the people purchasing these machines won't communicate to their staff that it's a flawless whodonit machine.

That's because the step from "potential leads" to "flawless machine" is a relatively small leap easily hidden by throwing out bad stats and case studies about people caught red handed. Even if you say the machine is 80% accurate, a lot of people will happily accept harming the 20% on the chance they might be in the 80%. (Though, I'm willing to bet the false identification percentages are both not public and not kept up to date, if they are even known).

> Telling someone wrongly that they are a thief, in public, seems like libel

Slander.

Libel is published, Slander is spoken. If they put a picture of her on a sheet that said "shoplifter" and posted in on the wall (or the net), it'd be libel.

But, I don't think you'd win a case based on Slander either:

* you'd still need to show it was published to a third party (though as you say, maybe you had friends present) * They can show that they reasonably believed it to be true (the system told them so). Their wording then becomes incredibly important - did they say "you're a shoplifter" or "you've been identified by our shoplifter recognition system"

They might also try to argue that stopping shoplifting is in the public interest and that this was a statement of opinion related tot hat.

I agree though, if this happened to me, I'd definitely be sending a letter or two to dissuade them from making a similar mistake in future (I think I'd also be trying my luck with Facewatch given that their system disseminated that "information")

Yes, I'd forgotten that libel is published information.

Although, given that this is third party tech, maybe Facewatch counts as the publisher.

Wouldn't there be a better outcome for those misidentified and publicly slandered and inconvenienced if they sued for stress? Some unlucky sod could have a heart attack from this.

Libel and slander must have malicious intent.

I think the better legal tactic would be defamation as it doesn’t require intent.

It seems that the proponents of this technology believe the benefits far outweigh the harms. Why not have a standard generous payout to anyone misidentified by the system. The payout would be funded by all parties involved, store, police, tech company. If the benefits really do outweigh the harms and the compensation is enough, I think most people would begin welcoming the program.

If the system got gamed (people with a history of false positives seeking to be misidentified), fine, it would encourage the technology to improve and eliminate false positives.

The payout would have to be at least enough to both fund and cover the inconvenience of moving to a new neighborhood closer to stores you aren't yet banned from. Of course, that becomes impossible if the tech is widespread enough.

It doesn’t really work like that in the UK. I doubt you’d get anyone to take it on, let alone for feee. Proving the material impact would be hard. She’s not been named publicly, no lost earnings etc. I would like to think this sort of thing means we get better regulation, but who am I kidding.

Reply on news.ycombinator.com