UK court clears post office staff convicted due to ‘corrupt data’

https://www.schneier.com/blog/archives/2018/04/securing_elec...

https://www.bbc.co.uk/news/uk-scotland-25639645

No doubt it's some sprawling, insane Java monstrosity Manhattan project or suchlike.

"Yeah - 10,000 classes - completely fine and not crazy at all."

Q) Did you or any of the people you got to examine the software found any way that what the defendents said was true?

A) No

Q) Then you are guilty beyond reasonable doubt.

I think the bigger issue here is around the power that a large organisation wields to duck and dive and use corporate tricks to manipulate how it played out. For example, the fact that so many people had been accused could have been analyzed if it was known e.g. Last year 5 convictions, this year, 700!

The most important part is that the PO used these actions to claw back "stolen" money from its postmasters. This money appears to have ended up in its profit and loss account.

If true this means that instead of the postmasters stealing from the PO, the PO was stealing from its postmasters.

There's been at least one claim - in the Daily Telegraph, so questionably credible, but never mind - that a document exists proving that senior management were aware that the accusations against postmasters were untrue, but carried on regardless.

If that document exists it changes the narrative from accidental tech failure and management incomprehension to something less wholesome.

This is what happens when you outsource core financial systems to low cost bidders with dubious tech chops building a message queue system is not fraking rocket science at this point.

Back when I worked on the ground up billing system for Telecom Gold (aka Dialcom) we did this as the existing mish mash of dodgy code that Dialcom offered (Sorry Eric) was not up to standard.

We had large amounts of internal auditing built in and we tracked discrepancies to the Penny.

The CEO blames one of their directors; the Director blames the supplier; the supplier blames the requirements documentation; the Business Analysts blame the culture for creating confusing and conflicting requirements.

Yes, you can hold the organisation accountable but then the people who worked there back then are long gone, they don't care if the Post Office gets fined £500M.

You only have to look at the enquiry into the flammable cladding scandal which was entirely down to fraud, yet, there are people who have not been arrested over their misrepresentation of their products.

ref: https://www.ingenia.org.uk/Ingenia/Articles/c05470e5-337f-4b...

Maybe the FCC went all NYPD-World-Police on the UK - popped over there to run things for a while.

Nationally regulated, sure. Verified with a physical copy (or a different system), sure. But banned altogether? You might as well ban everything in the world that is digital, as none of them are fool-proof.

Voting isn't even that important. The wrong guy gets picked, what happens? Same bullshit as if the right guy got picked. If your choices are "Hitler" or "Jesus", then your system is just fucked up, and making voting fool-proof isn't the way to fix it.

In addition, electronic voting would be a boon to democracy. It would provide another avenue for maligned minorities in remote areas be able to vote, when things like paper ballot voting in the middle of a pandemic might fail or be error-prone (esp. when a fascist fucks with the postal system), or local authorities enforce racist requirements like a physical ID card.

Damn straight. I'm really big on software Quality. It's kind of my driving passion.

It has been my experience, that an attitude of Quality is actively discouraged in today's "rush to a crappy, lashed-together-with-baling-wire-and-bandaids MVP" SV culture.

We glorify and make heroes of those that deliberately publish garbage, but make money doing so.

When we look to an industry to police itself; it never does. But the rules and regulations applied from non-domain-expert politicians are often ineffective, burdensome, and really only apply to a bygone era (See ISO 9001/CMMI).

Software doesn't exist in a vacuum and software will never be perfect. Trying to solve systematic problems by holding one part to impossible standards will just make things worse rather than better.

This drives me up the walls. At my last job (food ordering startup the CEO had the attitude that releasing code that was 95% functional was Okay, remaining issues could be fixed as we went along.

As a result, one developer overlooked a bug that cost the company €300,000, loyalty discounts weren't being deducted from payments to take-aways. They then had the cheek to demand take-aways pay them back.

Then they launched a major upgrade to the system at 5pm on a Friday - two hours before their busiest time of the week. It collapsed a few hours later and it was impossible to roll back because they didn't include a roll-back SQL script for the DB. It took till the following Tuesday to fix it.

The DB schema was all over the place and as a result it was slow. Entity Framework couldn't handle it and the SQL it was generating was terrible. Me being the only one with decent SQL knowledge had to replace all the bad EF queries with raw inline SQL.

Despite this, they still carried on deploying without a care in the world. I was told to stop moaning about QA. We didn't have QA or testing staff, the CEOs attitude being why pay for QA staff when our clients will do it for free?

Bob Martin talks about it a lot, how the software developers of the world need to have an "oath", like the hippocratic oath. Two posts that summarize things well (but there might be more where he talks about these things) https://blog.cleancoder.com/uncle-bob/2011/01/17/software-cr... https://blog.cleancoder.com/uncle-bob/2015/11/18/TheProgramm...

The phrase "move fast and break things" should be seen as cautionary, not aspirational.

The issue is very often related to massively complex corporate requirements (the Post Office makes me cringe, even today, with the complexity of their postal system) and then coupled with the ever-present need to keep costs low, especially when designing something so complex.

I doubt anyone building this thought it would be OK to break things!

I think people see a false dichotomy between making things quickly and making them safe. The fact is in the development of any complex thing, you're going to have bugs, and generally that's okay. But things should be designed to fail safe. Making something that throws errors when something unexpected happens is actually faster and easier than trying (and possibly failing) to handle edge cases; had Fujistu taken that simpler, easier approach then all this pain and suffering would have been avoided.

The key is how we respond when the software fails. The https://en.wikipedia.org/wiki/Therac-25 case shows an example of what not to do - when hospitals started reporting their machines giving lethal radiation doses to people, the manufacturer doubled down on the computers-are-infallible rhetoric, where they should have put every last effort into investigating. Likewise, the post office should have noticed that a rather excessive number of postmasters were apparently fiddling the books, and investigated. Instead, after it was fairly obvious that the computer was wrong, they pushed the computers-are-infallible line right through the courts, and that is what earned them the "affront to justice" judgment.

This won't change until executives go to jail.

A few years ago, we were fighting against tight deadline and skipping unit tests, QA, processes, etc. Someone brought up one of the recent major breach (Equifax?). Developers started to say that people will go to jail. Basically, devs were using this breach to imply that they will personal responsibility for releasing a product that might have security flaws. Our director laughed and said no one will go to jail and if our product ever got in trouble, they will personally take responsibility.

This is extremely domain dependent, and should be handled as such. And in some cases it already is - look at the testing / verification space shuttle code goes through vs your friends cat video side project website.

I disagree. The needed quality of your software changes dramatically depending on what it is used for. Something that helps someome semi-automate their workflow where bugs just mean they have to do things manually (as things that are wrong fail quickly and obviously) is something where what matters is speed of release and features. Something calculating how long people should be in prison needs to care about quality.

Anyway, the courts are the ones that have the biggest share of the blame here. Believing an unreliable witness for the prosecution is a common cause of injustice.

If the problem is the software, then use less software. Perhaps we shouldn't simply take it as a given that moving processes into software isn't always the right move?

And according to the article, the full number may actually be something like 900 people.

>Campaigners believe that as many as 900 operators, often known as subpostmasters, may have been prosecuted and convicted between 2000 and 2014.

How do you make this mistake almost 1000 times over 14 years before someone suspects the system data may not be quite right? Also, even if you do completely believe the data, how can you convict them all without additional supporting evidence, like new purchases that don't seem to fit their salary, suspicious bank transactions or balances, records of unusual system access or them actually manipulating data, etc.

I mean, I don't think anyone assumed they suddenly and inexplicably became thieves, just that the fancy new software finally caught people who have been scamming the post office for years. Obviously the software was completely wrong and it's criminal what happened to those people.

It took weeks to fix the problem.

[1] https://en.wikipedia.org/wiki/Robodebt_scheme

The real issue here was that Post Office refused to recognise that, although computers themselves are mostly infallible, computer programs are never infallible. They conducted their activities and took actions based on assuming the reporting was flawless.

Then the really serious problem is that in cases where the fallibility became more visible, they consistently and systematically covered it up and pressed forward with their incredibly aggressive enforcement work anyway, knowing how much damage it was doing.

This is unquestionably an issue of abuse of power and position.

I look forward to finding out if this was a “fraud system gone wrong” or a more basic ledger system failing to do sums correctly.

Partially addressing your question though, if you were to insert the words “AI” and “bias” into the sentence we as an industry are starting to figure this out. The certification and testing processes you mentioned are there in cases where a team’s mature enough to have both a data and model lifecycle worked out. You see words like MLOps trying to describe how to do that effectively in production.

For example, my work has both a design approach (in both the product design touchy/feely sense and software architecture sense) that includes questions and practices that will help to reason through data needed to address a problem, what can go wrong with that, and how things look when it goes wrong. The last bit is the most interesting one to me. In terms of practical engineering, inference results generally should have some sense of lineage - of data, model, and training services which explain how you got to a given answer, including what inputs were considered or ignored.

An interesting side topic with this is that poor implementations can result in inexcusable differences that affect downstream systems. For example, if a particular model has predicted something like “this transaction is suspected to be fraud” it better be consistent from run to run, and the input data better be consistent over time. If either of those changed - explaining that to the consumers of the data is essential to them understanding that either the model changed, the data changed, or both.

Corroborating evidence. In this case, where was the evidence that this money was ever in their possession? Was it ever sitting in their bank account? Was it buried in the back yard? Did they buy fancy sports cars or houses? The prospect of thousands of people stealing money without a trace of the cash is fantastical.

In general, I'd say electronic evidence should need to be corroborated with physical or other types of evidence to achieve a conviction. It's too easy for electronic records to be falsified, either through software bugs or outright malicious intent.

When people were unable to continue making up that shortfall this was seen as further evidence of their criminality: "they've spent the money", "they've hidden the money", and not "they never had the money".

In the UK in the wake of the 2008 banking crisis, a number of positions in banks became criminally liable for issues under them. If you're director-level or above (I think?) then you may be ultimately put in prison for negligence or issues like that which occur in your department. This is rare, not sure if it's been used yet, but it effected a cultural change in consumer banking as a bunch of execs suddenly had their necks on the line if someone under them did something wrong. I don't believe this is too hard-line in practice, I think a defence is "look at all these reasonable steps we take, we couldn't have foreseen this", but it had the impact (source, a good friend of mine is bordering on this level in a UK bank).

I wonder if a similar thing could work in a wider way across more industries - not with the intention of criminally punishing lots of people, but with the aim to change the culture around responsibility to the public and other stakeholders in the work that we do.

Not taking software results as a fact. Software report stating X in court should be equivalent to "the person who wrote this in a hurry would say X, but it's not a sworn testimony".

We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.

Corporate structure helps diffuse and deflect responsibility. Each group (executive leadership, middle management, and ICs) gets to diffuse and deflect responsibility and liability onto each other.

We already have all the positive incentives in the world - cash money. It's not enough.

Post office employee can’t afford a lawyer that would do any extra work

Earn enough so you can afford your rights... and appeals court where that actually matters :)

Well, same government first destroyed immigration papers and then deported and otherwise ruined the lives of their own citizens ( Windrush scandal ). I'd love to see the perpetrators in jail but fat chance.