Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation.
Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation.
To clarify, there was not _one_ bug, but hundreds.
https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr...
From my read of the information publicly available, there was a fundamental lack of distributed/transactional system understanding from the developers at Fujitsu.
This[1] article recently posted here[2] reminded me of the Post Office Scandal.
The lack of respect for established CS theory (transactions and distributed systems) and established accounting practices (double entry book keeping or even the idea of a ledger) is mind boggling.
[1] https://news.alvaroduran.com/p/engineers-do-not-get-to-make-...
And even if you do the right thing, if a competitor comes in and offers to do the "job" quicker and cheaper because they're doing the wrong thing. You may have the moral highground but that doesn't pay your employees' salaries.
What we need is a better framework for punishing bad software, in an ideal world without a bunch of red tape and reducing the burden on non experts identifying what is or isn't bad behaviour.
It's more likely that managers are just incompetent. Not being able to distinguish between infrastructure and "decoration" means all their decisions are at best superficially motivated.
The punishment we have for this is unemployability and bankruptcy. It will happen by itself if you let it.
A framework for "punishing bad software" sounds to me to be hiding the ambition to "protect bad managers from consequences of their decisions".
In my career, I have never seen a manager see consequences for making the decision that values "money now" vs "avoiding things going wrong later". I do not believe that it will happen by itself.
Imposing some formal framework is more likely to insulate managers even further while making software developers accept legal responsibility beyond their pay.
yeah, nah.
"No one ever got fired for buying IBM"
after decades of incompetence the multinational contracting firms are still going fine. IBM still runs fine and they're focus is now even more in the insulting side of the business.
So like engineering licensing and insurance?
It doesn't strike me as unreasonable, software engineers are now able to cause billions in damages and/or cause loss of life via primary or secondary effects.
Sure, one of software's greatest strengths was that anyone could learn it and it'll be sad to lose that but that's not really a reason for structurally enabling things like the Post Office scandal.
You probably don't need accreditation to serve cat pics but things dealing with money and life/death should. But IANAL so hopefully someone else can bring better insight to this area.
Software needs to be transparent, a human expert worker needs to be able to verify the result a software generates within reasonable time.
There is no expertise for software that you could delegate to aside from an entity using some form of integration test.
This is becoming more and more important as the AI industry is pushing non-deterministic computing further and further. There will be court cases where the full automated decision chain will be called into question, and a default assumption of "well, we don't have any logs so we should assume the software did the right thing" terrifies me.
Even speaking from an IC/non-managerial place, far too many technical problems in businesses are really just symptom-alleviation (or worse, performative look-I'm-doing-something theater) for a deeper problem which involves misaligned incentives for humans.
People will still be able to learn how to program and actually program. But if they take money or deal with people's private information, then they'll need to comply with the standards that will be regulated.
I think we need to start thinking of an individual's persona which includes all the information about them that is online or in government (non secret) files.
That needs to be considered when designing regarding regulations for software that interacts with someone's persona.
So anyone can set up the equivalent of an art stand in the park (serving cat pics), but if they start either selling cat pics or selling advertising that relies on collecting information about the personas on the site, then that needs to be regulated.
You can even have non-licensed people doing work, and a licensed individual signing off on the end result. They would need to review the designs and work to make sure they agree with how the work was done, because it's their license on the line if something goes wrong because of bad craftsmanship.
The law may require to use the ones that are certified if you are doing something sensitive. It doesn't need to be universal necessarily, but it should apply to public tenders for critical infrastructure at least.
Software isn't made of physical materials anyway, it's speech. How about instead we impose open source requirements to enable public verification of critical systems?
Some software is critical infrastructure and needs to be treated as such. We are not special. Every other engineering discipline has gone through this same process as and arrived at the inevitable conclusion that government regulation is essential, but only after causing unthinkable damage to the public first
I say we regulate the word engineer the same way it is in many countries for real engineering. if you don't want to progress beyond code monkey, you can be a software developer and innovate yet another react clone. if you want to be called an engineer, you learn and follow the regulations.
What they need to specify is the standards for software that certain types of organizations can use. Like government agencies, government contractors, medical organizations, construction and engineering firms, and probably some other kinds of large private businesses, depending on their industry.
Basically, if the software your organization uses can cause the level of destruction that Horizon did, it needs to have specific certifications, or you can't use it.
In order for such software to be certified, it needs to meet certain clearly-defined standards of quality, potentially including having all the technical leads of some level (or just all the developers, depending on various factors) be licensed, and have their licenses on the line of something like this scandal occurs.
It's not a panacea, and it would definitely be an absolute bear to get the terms of all of it defined both clearly and in a way that is likely to actually produce a quality product, but IMO it is likely to be worth it in the long haul.
Just mandate open source if using public money.
If someone prefers solution B to solution A, bringing up a situation that had neither is not a counterargument.
all it takes is enough people to die, and/or for rich people to lose enough money and it'll become the rule.
And it is not as absolute as you make it sound. Only dependencies for specific critical functions may be regulated. And they don’t have to literally force a whitelist of dependencies on you, just whichever has been certified as appropriate for that purpose.
[1]: https://en.wikipedia.org/wiki/FIPS_140
[2]: https://csrc.nist.gov/Projects/fips-140-3-transition-effort
Sure, now that the infrastructure for this has been built, it can be configured to require stronger crypto then FIPS does, but that infrastructure would never have been built without the likes of FIPS, and the government mandating it's use. And I know this because even with all of the hard engineering work done of building that infrastructure, there are no commonly used stronger policies; because the only people who actually care are the ones forced to care by the likes of FIPS.
Our electrical standards might not the safest way of wiring buildings, and not what we would come up with if we wrote the standards today. But they are orders of magnitude safer then what electricians would be doing without the standards.
What prevents regulatory capture?
Food, drugs, healthcare, consumer products, chemicals, cars, planes, trains, buildings, utilities, energy, infrastructure, salaries, loans, investments, accounting... Even media requires some licenses, receives age ratings, and has restrictions on advertising.
It's not rocket science, this is normal for every single other industry.
Rocket science is one of the few industries that's actually seeing active innovation.
Perhaps without the certifications lots more people would have died. I'm just an armchair analyst. Just food for thought.
Is Fujitsu run by engineers?
> So like engineering licensing and insurance?
How is engineering licensing and insurance punishing bad software ?
Microsoft is still going strong and all they do is "checklist security".
The number one rule for engineering domain applications is to understand the domain.
I would blame the perennial neophilia and lack of (or inadequate adoption of/respect for) standardized texts in the industry. Though, to be fair, a lot of this does come down to the rapid changes in the technology.
Iterative development is necessary for software, of course, but this should be understood as a necessity due to the medium, not as an excuse for skipping research and design. A lot of these domains (especially something as critical as accounting) should be solved problems.
No doubt management also oversaw the development of the system and rushed it to production.
A healthy culture should accept failure as inevitable and learn from it when it occurs. It should also listen to the people who know best: the engineers who built the thing. You know, like the aerospace industry.
The damage that morons in suits do in pursuit of their bonus cannot be overstated.
Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy.
https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr...
Transaction idempotency is such a basic property for a financial system that I struggle to believe that Horizon was tested in any meaningful way.
"the engineers who built the thing" (Gareth Jenkins) are also under investigation for perjury.
[1] Horizon is still in use, in its buggy state, with replacement scheduled for 2030....
The developers were just doing their job. It's management's responsibility to construct a functioning system of checks and balances and understand the limitations of their systems, both of which they failed to do. If it weren't for their hubris the fundamental issues with Horizon could have come to light much earlier.
Let's also not forget that the reason executives are compensated well is for them to take accountability in situations like this.
More broadly, your idea that this is solely a management problem is how we end in situations where developers are being told to unquestioningly code some design exactly as given, which never works. You don’t get professional judgement if you don’t accept responsibility, too.
But I would also say that that kind of toxic management is absolutely a part of "software engineering culture". How many horror stories do people on here have of managers who care nothing about the quality of the product, only meeting the deadline so they can get their bonus?
"Software engineering culture" is way, way more than just "how write good code." It includes how we work, how we manage/are managed, how we advocate for ourselves, or fail to do so, and much more.
It certainly includes the very common resistance to unions among programmers, and assuming this was caused by management pushing a known-bad product out the door, a strong union would have (at least potentially) been able to stand up to such demands.
The thing literally went into a criminal court as evidence, and was "presumed correct" in a way that overloaded any technical or reasonable discordance.
The largest failure here was from the judges and lawyers. The software failure isn't even relevant.
Where signing off contracts does have some implications beyond "it works on my computer".
In al seriousness, Engineering is about verifying systems to make sure their lifespans and failure modes are known, up front.
This has a legal dimension and a practical one. Legally you can make people liable for unreliable systems. But you can also be liable for failure to maintain properly, or failure to warn about impending calamity. Because it's all documented and verified.
Practical you can live worry free in earthquake and flooding proof buildings, trusting in the diligence of Engineers, and maintenance workers, because they and others have liability imposed on them.
For software this is only the case in a few sectors. For buildings in all cases. Not comparable.
Any life can be absurdly destroyed via malware, security exploits, accounting gone wrong, a database deleted in production,....
People also don't put up with faulty products, why should computing be an exception, shitty ship now fix later culture?
And above all, calling oneself "engineer" out of a bootcamp, has nothing to do with Engineering.
Try suggesting to use a tool like TLA+ to validate some complex design and the most likely scenario is that people will laugh at you, even if it's a critical component for the business.
Most decisions in the industry are based on weak anecdotes and unfounded opinions of underserved "authorities".
"Majority of Subpostmasters still getting unexplained Horizon discrepancies" - https://www.postofficescandal.uk/post/majority-of-subpostmas...
One of the things the Post Office did was sell travel money, but the whole system was never really designed for ForEx operations, so it didn't keep track of exchange rates over time. The result is that reconciliation used the exchange rate at time of reconciliation instead of at the time of trade. So, if the foreign currency had gone up in value, it would show up as GBP missing.
Given the Post Office had a reputation for really good exchange rates, this one design flaw might be responsible for a significant portion of the problem.
Wow, that's... dumb.
The main one is that the post office management/officials at some point became aware of the bugs and that they were ruining lives of innocent people and they knowingly kept lying to save their asses.
The inquiry page has all of this and more: https://www.postofficehorizoninquiry.org.uk/about-inquiry
I’ve read through it, it’s long but it’s a good (while terrifying) read.
And wiki has rest it the judgments: https://en.m.wikipedia.org/wiki/Bates_%26_Others_v_Post_Offi...
I have a lot of respect for the judge after reading this. Here’s a quote describing Post Office evidence:
“bare assertions and denials that ignore what has actually occurred… [amounting] to the 21st century equivalent of maintaining that the earth is flat”.Both the gov and Fujitsu and the post office absolutely knew it was a bug and intentionally hid the fact while the post masters lives were ruined.
However bad the bug was, the cover up was _much_ worse.
https://www.computerweekly.com/news/366587174/Fujitsu-set-fo...
The real scandal here is that there were hundreds, and those at the top knew this, but instead doubled down.
I am pretty sure more dependable systems send the odd person to jail - there have been many cases where someone was lucky not to be convicted.
There's a little more to it. Most of the comments here are focusing on "correctness". And yes, the amendments to section 69 do something towards tempering its ridiculous and dangerous "presumptions".
But the (UK Post-Office) story is that the Horizon system had back-doors in it. Fujitsu denied this. "Corrections" were made to systems without operator knowledge - to fix actual errors caused by a terrible database sync script full of race-hazards, faulty locks and duplicated state.
The cover-up began life as engineers trying to hide up technical mistakes. It escalated to senior executives trying to cover up financial and political mistakes. It ended with the Crown colluding in covering up judicial mistakes. It is an exemplar of hubris, pride and egotsim resting on a refusal to give up a religious belief in technology. Were it not for the courage of a few (including judges and MPs) they would have gotten away with it (if it weren't for those meddling kids)
The case stands as an important landmark that you cannot "hide behind" technology as a means for abuse and injustice.
The amendments are welcome but insufficient. They open up a good opportunity for cybersecurity people to work with lawyers now.
There are two outstanding problems:
Proprietary code. If you cannot examine the system then the right to challenge it is meaningless. This requires changes to investigatory powers/discovery if anyone wants to use "technical correctness" as a base for argument.
Malicious function. While the discussion revolves around correctness it is incomplete. Many systems (perhaps not the Horizon system) are not faulty, they work perfectly well to deceive, manipulate and swindle.
I'd still push for a complete reversal of presumption [0]. Where software is part of a legal dispute it should "take the stand" as its own witness, in that formal proofs of correctness (a very VERY high bar in software engineering) need to be brought in front of the court. Otherwise the reasonable presumption is that an error or hidden malicious coding "cannot be ruled out".
[0] https://cybershow.uk/episodes.php?id=23 https://cybershow.uk/episodes.php?id=24
I think it would leave me furious for weeks and push me into a depression rut.
I'm trying to take the whole debacle as long term inspiration to be excellent at what I do - it's tough to stay positive when so many people involved (Fujitsu, PO upper management, the original prosecution) seem entirely morally bereft, with little chance of consequence.
When you have such a widespread and ongoing problem it becomes clear that such a large proportion of postmasters can't be criminals. Computer bugs are well known. Eventually it is going to come out that the computer is wrong. Why double-down? The earlier you admit a mistake and apologize the lower the impact and the less anyone cares. They had a built-in scapegoat that everyone understands and accepts: the vendor's software had bugs! We have daily meetings to yell at them to fix the bugs we promise and we will fix the problems ASAP.
Or do the sneaky thing and fix the issues, stop prosecuting postmasters, and ignore the ones you prosecuted by mistake. Cynical, cruel, and immoral... but contains the damage.
Instead the UK Postoffice seems to have just let the problems continue while simultaneously allowing prosecutions to go ahead knowing they were faulty. Literally the worst of all worlds: ongoing accumulation of liability, now with provable malice!
I heard some convictions of innocent people have been overturned, or are in the process of. But new convictions of the actually guilty... no news about any such thing. Maybe I'm just uninformed.
Fair enough. The investigation is still ongoing but as of now I don't think anyone who knew (or should have known) has been punished. The UK public are the ones on the hook for recompense.
As low as it takes for a profit!
Whether the story was true or not, the government is just bad at science. It's how we get stuff like bite mark patterns and facial recognition warrants.
Yes there is operational separation but it's not like they're wholly unrelated and the government totally guilt free