AFAIR, the breathalyser was incorrectly averaging the readings, giving disproportional weight to the first reading.
I don't know if it was enough to rule in their favour, but I'm sure it called the data into question
Edit: Looks like it was a Draeger breathalyser https://www.schneier.com/blog/archives/2009/05/software_prob...
As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.
I don't see those particular issues make it biased, just inaccurate - it could go either way.
Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store.
The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on.
The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest.
The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.
There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried.
I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.
What is this based on?
The somewhat-less-malicious interpretation is that the companies have a strong incentive to detect + fix errors that cost them money. Meanwhile, consumers are a) non-centralized, uncoordinated, and often unaware of errors, and b) have no way to fix systemic issues that impact them. And the companies therefore have no /real/ incentive to fix systemic problems. It is literally more profitable to fix the bills of the few people who complain, as they still make money on the remainder who don't notice the errors in the first place.
(on edit; exactly what the other comment one subthread over said. :P )
This could just as easily be selection bias: the errors in favour of the customer are less likely to get reported by customers.
i.e. The initial error may be randomly distributed. But the follow-up on the error will have a lot of bias.
https://www.nytimes.com/2013/10/26/business/toyota-agrees-to....
Also let's remember that a company in UK was selling fake bomb detectors to Israeli and other militaries, and it took them more than 10 years to notice!
https://www.bbc.co.uk/news/uk-29459896
There needs to be proper scrutiny into these things, I could start some random 'deep learning to find criminals' company tomorrow, and have less regulation than a car mechanic
https://en.wikipedia.org/wiki/Sudden_unintended_acceleration
Occam's razer points to people just hitting the wrong pedal or people's floor mats getting stuck.
The reason Toyota ate it in the press for this was competitive.
can you explain what you mean? I don't understand this sentence.
The reasons are manifold, including:
- Normalized values need to be averaged differently the absolute values.
- Floating point has limited precision, even just correctly summing/multiplying numbers need special care if you care about correctness. Results can, in the worst case, be of by a massive amount.
Often you don't need to care about it so it's not uncommon for especially junior programmers to be not so aware about it.
I mean in the last 3 years of working as a professional software engineer/developer I didn't need any of this at all, but once I do I know what to look out for.
I know, this is exceedingly cynical.
No, not really.