> A tool designed to find people guilty is biased to find people guilty.
I don't see those particular issues make it biased, just inaccurate - it could go either way.
I don't see those particular issues make it biased, just inaccurate - it could go either way.
Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
This could just as easily be selection bias: the errors in favour of the customer are less likely to get reported by customers.
What is this based on?
The somewhat-less-malicious interpretation is that the companies have a strong incentive to detect + fix errors that cost them money. Meanwhile, consumers are a) non-centralized, uncoordinated, and often unaware of errors, and b) have no way to fix systemic issues that impact them. And the companies therefore have no /real/ incentive to fix systemic problems. It is literally more profitable to fix the bills of the few people who complain, as they still make money on the remainder who don't notice the errors in the first place.
(on edit; exactly what the other comment one subthread over said. :P )
Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store.
The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on.
The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest.
The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.
There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried.
I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.
i.e. The initial error may be randomly distributed. But the follow-up on the error will have a lot of bias.