Hacker News Reader

Comment by mcv | Hacker News Reader

Summary: the presumption that computers are always correct had lead the UK post office to believe that hundreds of post offices had shortfalls in their accounts, leading to hundreds of postmasters getting accused of fraud, resulting in their prosecution and conviction.

Fortunately, group action has lead to the error in the system being discovered and their convictions being overturned.

It seems pretty dangerous to blindly assume that a computer system is always correct without thorough proof that it is indeed correct. But we already knew that.

View on HN

So all criminal prosecutions involving computers should require the state to prove that the computer is correct?

A complete proof of correctness might be a bit much to ask, but slightly more burden of proof than "it's what the computer says" would be nice. Every piece of software of any meaningful size has bugs. First check if the discrepancy might be the result of a bug before you start sending people to prison.

Why not?

Think of it in the other direction, why should you be allowed to use evidence that has no documented reason of being accurate.

It's not possible to prove, without billions of dollars of investigation per device, that any complex consumer electronic device is operating 100% correctly.

So wouldn't it practically invalidate any evidence that was touched by any device?

Why would it cost billions of dollars to satisfy the court of the accuracy of a program’s claims when it doesn’t cost billions of dollars to satisfy the court of the accuracy of an investigator’s claims?

This is presuming they expect 100% confidence.

If they are fine with a lower bound of confidence, then that's already the current situation.

Courts don't bother themselves with confidence intervals nor do they bother with 100% ("reasonable doubt" is not 0/100%).

The problem is the current situation. Evidence is being used without actual proof that said evidence was collected "truthfully". An algorithm will generate a result but if you haven't proven/shown that the algorithm generates a correct result then any result from that algorithm shouldn't be usable in court. This problem comes up a lot with people who (un)cleverly use NULL for things (i.e. License Plates). Or if you live in the middle of America and somebody decides your address will be the default for geo-ip lookups if none is found.

To stick with the article, immediately when the system was deployed the postmasters thought something was wrong. If the state had to show that the computer was correct the whole situation would've been avoided.

No sudden unexplainable wealth, no bank transactions to personal accounts, no incriminating text messages. Even if the code was correct that alone is not proof the individuals accused took the money, but they were still convicted. That isn’t supposed to happen.

For the postmaster example, they did show it to be correct at their accepted level of confidence. Which is that it returns superficially believable results.

100% correctness for such a system would certainly cost many billions.

If you mean that they should have had a higher bar, then sure, but that's a political choice of how much effort to invest, not a technical one.

Is any aspect of any legal system anywhere based on 100% confidence?

"Oh, but the computer is doing the right more often than (9.4608e+16)-1 times out of 9.4608e+16 times!"*

Yes.. but that's "less than" a single mistake per year, per computer. How many cores do your data go through per year ?

* Totally made up, rough number of cpu ticks of a 3 ghz core in a year.

Still, the point I'm trying to make is that with the mind-numbing number of actions carried out by another mind-numbing number of machines on a third absolutely mind-numbing number of bits every year, you will need a metric fuckton of zeroes after your decimal point, before it's "not going to put innocent people in jail".

If it's metric, it's a fucktonne, not a fuckton.

‹ducks›

Interesting, didn't know that, thanks! :)

In Danish we use the word "Ton" which is 1000 Kg, didn't realized that English has two words for it, didn't realize there's an emperial ton either :D

Yep, US customary ton is 2000lb

Of course… we all know how buggy all computer programs are.

I don't think it is an either or question. But it seems very reasonable to not assume computers are always correct.

In a situation in which a bug could have conceivably caused the evidence against the defendant the defandant, should be allowed to examine the computer system (read the source code). Without that I see no reason why it should be allowed as evidence against someone, when the evidence could have been the result of a flaw in the software.

Of course "could" is pretty ambigous. Things you might look at are: Have this or other, similar systems suffered from a bug which could have caused this? How much/often is the claim out of line with what the defence claims. How serious and easy to notice would a bug have to be to cause this? Is there other evidence, besides the computer? And probably many more. Of course "the camera changed the face of the person to the defendants" is something which probabbly should be disregarded.

Prior to that I read of bogus prosecutions based on ATM withdraws.

Reply on news.ycombinator.com