Fixing Horizon bugs would have been too costly, Post Office inquiry told
theguardian.com
theguardian.com
Right now on YouTube there are Fujitsu staff being asked questions by the enquiry on this topic.
Worth also saying that some bugs were discovered in a product which was due to be replaced by a new system so it was considered low priority to fix. However it impacted cases of several years before it was discovered.
I think maybe that 2 Fujitsu employees are either investigation or being currently prosecuted by the met police, so there are criminal proceedings being under taken right now in relation (I guess) to these bugs in the older version of the code. There's not much info about this currently for obvious legal reasons.
Maybe at first they must have thought that these were one-off issues and that they could just prosecute one or two postmasters, sweep the whole thing under the rug and forget about it. But that doesn't explain how they continued prosecutions into the hundreds years down the road. Like what did they think the endgame here was? Just prosecute indefinitely to cover up for the system?
I'm not defending the post office, but I suspect you have to look at it in context.
My understanding is that during that period of time (1999–2002) the Post Office's accounts took two large hits:
- £570m write-off of the failed Horizon project
- £1bn losses from the so-called renewal plan
Instead Ed Davey doubled down on the lying and persecuting ordinary postmasters, over an issue that until then he had no personal accountability for. This is what democracy is for, to provide the opportunity for a reset, to change course, and a mechanism for accountability of those who made the mistakes. Instead he allowed himself to be co-opted by the system.
https://www.telegraph.co.uk/news/2024/01/10/post-office-exec...
The Post Office was a money-hole; it hadn't made money since privatisation. It was effectively still a government agency.
UK government agencies have a long history with ICL, the UK "champion" mainframe maker, which was taken over by Fujitsu. It appears that this "champion" status was inherited by Fujitsu.
ICL couldn't compete with companies like IBM on merits; they had to have backdoor support from politicians and the civil service. I don't know how that worked; but if MI5, the armed services, the National Grid and so on were reliant on ICL, then it's understandable that government would want to encourage agencies to buy ICL.
So I suspect that the PO were pressurised by ministers and civil servants to buy Horizon from ICL/Fujitsu, and further pressurised to keep schtumm about the defects. Well, that's my guess: the higher-ups owed their jobs and reputations to ministers and civil servants, because they were effectively employed by a government agency, and it was losing money, and the government wanted to protect the reputation of ICL/Fujitsu.
All in all its a lesson in just how easy it is to cover your arse as an executive in a large company, and get away with pretty much anything. Only the most tenacious work by many people over decades has any chance of holding you accountable.
[1] https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...
Here is an anecdote from a past job.
1. Years ago (decades), someone made the contract rules. They did it in an ad hoc not very systematic way. They assumed everything would be judged by a human and be judged reasonably.
2. That person left and the department expanded.The rules had to become more formalized, but still room for plenty of exceptions. An exception could be a penciled scribble in the margins in a file or later on, a sticky note.
3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed.
4. A business analyst with no real understanding of the system is hired to generate requirements for digitization. The current overseer of the rules, someone who doesn’t use a computer much (yes, these people still exist) kind of just nods every time he creates these tremendous flow charts. this is the first degradation of knowledge. Even then though, this business analyst is not technical or legally trained, so doesn't appreciate why defining things like "end of day" or "contract billing cycle" precisely is crucial. even if he could, many contracts had sticky note individual definitions.
5. The business analyst goes back and feeds the information into the ticket mill of Scrum. To be nice and agile, work is done in bite sized increments with no regard for context. This is the second knowledge degradation. All those flow charts are chopped into little boxes. Devs see ambiguities, but have no interest in pushing back.
6. Developers only have the tickets to work on and quotas to meet, so write the code as requested. Testing is another set of people, so code is just tossed over to QA. QA has no more context than dev, so if it matches the ticket, they approve it.
6a. Nobody budgeted for automated testing so things break over time for lack of tests. Product Owner wouldn't grasp the concept, business analyst had a budget and wanted to deliver more features, and devs didn't want to fight about it.
7. Final increment goes back to product owner who doesn't understand and just nods. A piece of software that doesn't work all that well gets approved. Repeat for years and constantly adding post-it note rules as code and you have a system that has lots of errors.
A sane system would be 'contract A type' and all other be done manually or something.
Er....shit in. Shit out?
A great example of something hard to digitise is mechanisms for reward/punishment. Algorithmic approaches for criminal sentencing that work on a parameterised basis have been proposed over the years but always found wanting.
That said, it was certainly shoddy
'Shoddy', being a euphemism for 'beyond knowingly-being fucked-up, so much so that we'll cover it up oh fuck we're still alive this wasnt what we were told would happen'?
I dunno, or maybe just me surmising.
<<I wish I was joking>>>
Yes, you're probably right. But this is the part that really bugs me :
"The inquiry heard that in 2008, a glitch in a system called CABSProcess, which automatically summarises a post office’s transactions at around 7pm daily, resulted in users working at the same time having balancing issues."
This, for an accounting system is just fucking inexcusable!
I worked on a huge accounting project for a financial institution processing 2M account bookings per day (at that time, it's a lot more now). A lot of effort was put into the audit log (every booking was written to an append only log) and into the reconciliation process. When it was 20 Rappen off there was digging until the inconsistency was found.
That transactions are not completely isolated when that CABSProcess kicked in is just completely and uttelry inexcusable.
Not sure a sometimes offline distributed database where data is temp stored on POS machines is such an easy thing to write either
As usual: rushed, not well tested, assumed to be perfect
Certainly there were bugs that made it seem like there was more money coming in than actually did, but it seems like also there were capabilities in the platform that would also allow editing records to make it seem like there was actually less money coming in, which would enable skimming money out of the system almost (?) invisibly.
I also don't think this capability was ever suggested anywhere as being used in conjunction with a postmaster to skim money out. Maybe its possible in theory?
Management are motivated by short term bonuses.
The biggest shareholders are fund managers who get judged on their annual performance (In the UK there are annual performance tables published. I assume it is similar in other countries) so they have a short term outlook as well.
Private equity have a slightly longer term view, but not much more so.
No one has a stake in what happens a decade from now.
But now, in the end, the companies themselves will just get a financial fine at some point, not even matching what it would have costed them to do the change. And so no real consequence, no change...
We're all worn down by this and, in essence, we operate on blood money.
Surely it ought to be: pay back all 10 million, PLUS compensation to the people from who you illegally got money, PLUS a fine, PLUS holding the actual people making the decision (because surprise, a company is just actual real people doing stuff) at the very least financially accountable.
If you look at the version history of any software, you'll find it's full of bug fixes. Writing software without bugs is literally impossible. It's incredibly complex and written by "actual real people" and that means there will be mistakes.
If those mistakes lead to millions in liability, then you're going to have to start paying me those millions upfront, before I write a single line of code.
It's the same with a CEO - if people are criminally liable for mistakes made by people working under them... then nobody's ever going to accept a job where they delegate work to other people.
The solution to this issue is not to attack people who made a mistake. It's to put systems in place where you check for mistakes. I guarantee you the jury and courts that found these people guilty were presented with "facts" such as "nobody else can remotely access the software" which, it turns out, were total bullshit. The people who presented those facts in court are the ones to blame for this. They clearly either lied or made assumptions.
Both of those are serious crimes. You don't say anything in court unless you actually know it yourself. If someone else tells you something is true, you don't repeat what they said. You bring those people in as an expert witness to talk to the judge/jury directly, under oath, and you tell them not to make any assumptions either.
There should have been millions spent on a full code audit done checking for bugs in the software before allowing it to be admitted as evidence in court.
But knowing this and lying about it, as was done by the Postal Service, Fujitsu management and insurance companies, most definitely should.
You shouldn’t be liable for mistakes, but you should be liable for malcontent
> But a saddening thing is that if it were not those particular identifiable individuals who were culpable (and they certainly should be held to account) then it would have been other individuals doing the same things. And this is because of legal and corporate contexts that facilitated this wrongdoing.
> This is not to say that the guilty people can themselves blame the system - the culpable individuals could and should have done different things, at each and every step. Indeed, exceptional individuals could have stopped the nonsense and the cruelty.
> But these were not exceptional individuals - they were individuals doing what they (wrongly) believed to be their job or performing what they (wrongly) believed to be their function or protecting what they (wrongly) saw to be legitimate interests.
Those people are culpable for their actions, but they did so in a system that made doing the wrong thing easy, and such shortcomings will lead to another case like this.
Just for the “but they’re a private company” crowd: the company is wholly owned by the government and there is a minister whose responsibilities include postal affairs which includes the Post Office