This case was brought to public attention and repairs were attempted only because it's huge and involved hundreds or thousands of people.
How many disparate cases there are, where people's lives are destroyed and innocents are rotting in jails, we have to ask?
The legal system failed these people horribly. And the people who pursued these cases with no direct evidence whatsoever should suffer jail time.
However, the role of software here must not be minimized. Software makes it easier than ever to diffuse responsibility and create opaque processes that leave the least powerful people at the bottom of the hierarchy holding the bag. By rigidly encoding flawed assumptions and executing them without question, software is the ultimate realizer of our Kafkaesque nightmares.
No argument that the software bears fault, too.
However, when you accuse someone of stealing money, you should have to prove that they stole the money. This isn't some invisible crime. There should need to be evidence that the stolen money went into their account, got spent to buy something, got pulled from the till on camera, got transferred to Bitcoin--something.
The fact that all these people got convicted with no evidence that the money was ever in their possession is a gigantic legal problem.
Judges, salesman, and managers don't understand that.
I think for the courts the issue is a bit more subtle. The question is, who's job is it to prove that the other person is wrong ("burden of proof")? Should it be the job of the prosecutor to prove that Intel's processor produces the right answer when an ADD instruction is executed? Or should it be the job of the defendant to show that Intel's processor doesn't produce the right answer? What about proving that the compiler produced binaries which faithfully represent the algorithm? What about Excel?
In our normal life, if a computer is doing the wrong thing, we don't start by assuming a broken compiler; we start by assuming that the new, not-well-tested code is probably broken.
It seems that in the UK before the 90's, the burden of proof was always on the prosecutor to prove almost everything about the system, which is kind of ridiculous. So they passed a law trying to fix it, but messed it up the other way, putting the entire burden of proof on the defendant, without giving them any real way to disprove it. (I mean, shouldn't "discovery" at least mean I can inspect the source code?)
A more balanced law would say that widely-used software with extensive test suites can generally be assumed to be working properly; but that custom-purpose software needs at least some level of evidence that it's correct, and that defendants have a right to inspect any software that's used against them in court for defects.
The skillfull programmer may accept liability when you give him a verification team with a few PhDs, the ability to withhold signoffs, flexible deadlines etc. etc. Few are willing or required to pay for that. So they get a mystery box with a 90% chance of crap.
Laypeople generally understand that software may crap the bed in the sense of "the system is down, please wait, then try again". But few people have experienced subtle changes in stored data.
A judge looking into his document cloud may be ready to see a "sorry, not available right now" notice, but doesn't expect that some sinister program is, in the background, silently editing texts of his judgments and pronouncing people guilty when he intended to free them etc.
The problem with the Horizon scandal is in this sinister manipulation of data. It may also have been done by Fujitsu people themselves, in order to cover some tracks and tamper with evidence. This is a very untypical failure mode.
"We need some software, ok let's get a big reputable company in to do it for us, we shouldn't get bogged down with all those horrible technical details"
They do understand that. They do not care.
I disagree, the software glitch was the problem here.
We are supposed to be able to rely on computers to store and add numbers or report a system failure. This accounting software showed in black and white that some funds that the sub-postmasters were responsible for had gone missing.
What else was the legal system supposed to do? The broken software was simulating crime perfectly.
Except it wasn't; the main problem was how the PO was handling it. ICL/Fujitsu were aware of near-identical bugs in an earlier project[1], and PO employees omitted parts of an audit from 2004 that described similar issues as well[2]
It all goes back to ICL/Fujitsu and the PO being aware of the issue and withholding the information from anyone not already "in the know"; lawyers, judges, changing witness statements to hide incriminating evidence, etc.