Hacker News Reader

Comment by PaulKeeble | Hacker News Reader

Software is in the walls. At some point legislators are going to come and ask the question how we stop things like this happening and if the Fujitsu's of the world don't have an answer then we can expect regulation that will likely embed practices that don't help.

I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives. Things must change. This isn't a unique issue to Fujitsu it is something most of the software industry is doing, this story could be about just about any piece of software.

View on HN

> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives. Things must change. This isn't a unique issue to Fujitsu it is something most of the software industry is doing, this story could be about just about any piece of software.

Damn straight. I'm really big on software Quality. It's kind of my driving passion.

It has been my experience, that an attitude of Quality is actively discouraged in today's "rush to a crappy, lashed-together-with-baling-wire-and-bandaids MVP" SV culture.

We glorify and make heroes of those that deliberately publish garbage, but make money doing so.

When we look to an industry to police itself; it never does. But the rules and regulations applied from non-domain-expert politicians are often ineffective, burdensome, and really only apply to a bygone era (See ISO 9001/CMMI).

It's been 15 years since I've seen CMMI mentioned, and I was glad I hadn't.

Do you object to CMM as a model or just the CMMI/9001 industry?

Both. It's very well geared towards maintaining a certain standard of quality and predictable project throughput in rather well defined projects, and it certainly makes the job easier in procurement, but it's totally detached from what the SWE world is outside of that.

I can see the point of such models in certain areas, like military, aerospace, naval, or, to stay on topic, Horizon, where dev is outsourced, somewhat critical, specs rather set in stone, and non experts need to measure how capable an organization is to deliver, but for anything else it just feels like unnecessary meta-management that brings significant organisational and development overhead.

They had a good idea, but they applied "old world" thinking to it.

The single biggest issue with software development, is that it is incredibly dynamic.

Static solutions don't work, and CMMI is a very static solution. Sadly, a lot of quality practices are static.

Dynamic solutions are really difficult to get right, and tend to depend on a lot of hard-to-quantify variables, like the experience and talents of individuals on a team.

For example, I am quite good at designing fairly complex systems, as long as I am doing it alone. I can hold some fairly ambitious designs in my head; which allows me a great deal of flexibility. I can start with a fairly "fuzzy" architectural model (I call it my "napkin sketch"), and begin a project fairly quickly. As the project progresses, I can apply some massive structural changes, and pivot fairly easily.

However, the minute I need to communicate this plan, the whole shooting match comes to a screeching halt.

Team overhead is a really big deal, and I believe it is seldom factored into our plans, in any kind of realistic manner.

Maybe don't use an "incredibly dynamic system" as evidence in criminal cases, then.

Yup.

AI is gonna pour rocket fuel on this stuff. There's already a great deal of talk about replacing lawyers with AI.

Actually there's a huge amount of self policing. Engineers are the ones at the forefront of inventing and tooling more ways to test.

Yes, and no.

We have some marvelous CI/D tools at hand, but the execs are the ones that push to release before ripe, and they won't let things like auto-test failures get in the way of MVP.

There was a comment here, some time ago, that was made by someone that proclaimed themselves to have started and successfully exited a number of companies. It went something like "If you do not get physically sick, looking at the code in your MVP, you are spending too much time, worried about code quality."

I think that's a pretty good summary of today's startup zeitgeist.

That sounds like bad engineers. Set your CI/CD to fail to merge code if any automated tests fails. We're not talking weeks of extra work to get tests to pass.

I think that as soon as we say "zeitgeist" we're abandoning all attempts at understanding the full picture. There are huge numbers of tests being written all over the place, particularly with modern software tooling. It's the older stuff (e.g. Oracle Forms) or low code stuff (e.g. Bubble, Dynamics) that are hard to test where the biggest issues crop up, IME, as those technologies will be picked by people who don't value testing.

The main issue is management; not engineering. I'm actually quite chuffed at some of the amazing tools I have at my disposal, these days (like profilers and code coverage tools, as well as runtime debuggers). I "grew up" in tech from the early 1980s, and things have changed.

The one thing I miss are ICEs (In-Circuit Emulators). Those were badass, but processors are so massive, and so fast, that a real ICE would probably cost a couple of million dollars, and be out of date by the time it hit the market.

Tech can't fix bad managers, and money is like Miracle-Gro™ for bad management, if it incentivizes rotten quality.

Obligatory MonkeyUser: https://www.monkeyuser.com/2021/introduction/

I think the bugs left in by the SV culture are less important than the one they do fix.

The most important bug is that the software doesn't solve the problem that you have. It doesn't matter how reliably it doesn't solve your problem

> The most important bug is that the software doesn't solve the problem that you have.

And we should add:

Unless we can't do so without introducing any additional problems, while solving that problem in a manner that truly solves it; as opposed to making it appear solved.

We really are often best off, with the problem, if the cure is worse than the disease.

When I was younger, we had a saying:

To err is human, but it takes a computer to really fuck things up.

This seems more an issue of bureaucratic incentives than software. Fujitsu wanted to hide bugs to look better for future contracts. The Post Office wanted to hide bugs to deflect blame from central leadership and be able to scapegoat people at will. The judicial systems seems to have either not cared or had incentives for some quick prosecutions.

Software doesn't exist in a vacuum and software will never be perfect. Trying to solve systematic problems by holding one part to impossible standards will just make things worse rather than better.

Yes, it seems clear that people knew about these problems — they were obvious at one level of management — and they worked together to cover them up.

Heads have to start rolling for this, or we will end up in a dystopian nightmare where any corporate organisation can ruin your life for no reason.

It will be like USSR except more unpredictable because it can come from any direction

I think we're already over the edge of that, it's more urgent than you think.

A couple of insensitive Facebook posts gets you dropped from consideration for a job... no matter how long ago and how much you may have matured in the meantime.

Google implements FLoC and cohorts start identifying political leanings, medical conditions, mental health issues, anything that's legally potentially discrimination territory... how do you know that someone deduced a cohort topic and denied you <something> based on that...

Tip of the iceberg. Data aggregators already have opaque records on probably everybody alive, just find the one with data about your person of interest.

This needs to be a complete change of awareness and ethics and global law... otherwise we're going to have the movie "The Circle" come completely true as opposed to being just around the corner.

> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things.

This drives me up the walls. At my last job (food ordering startup the CEO had the attitude that releasing code that was 95% functional was Okay, remaining issues could be fixed as we went along.

As a result, one developer overlooked a bug that cost the company €300,000, loyalty discounts weren't being deducted from payments to take-aways. They then had the cheek to demand take-aways pay them back.

Then they launched a major upgrade to the system at 5pm on a Friday - two hours before their busiest time of the week. It collapsed a few hours later and it was impossible to roll back because they didn't include a roll-back SQL script for the DB. It took till the following Tuesday to fix it.

The DB schema was all over the place and as a result it was slow. Entity Framework couldn't handle it and the SQL it was generating was terrible. Me being the only one with decent SQL knowledge had to replace all the bad EF queries with raw inline SQL.

Despite this, they still carried on deploying without a care in the world. I was told to stop moaning about QA. We didn't have QA or testing staff, the CEOs attitude being why pay for QA staff when our clients will do it for free?

I worked at a place that internal customers complained QA took too long, so IT said fine, we won't do any. Then they complained the software didn't work... people sure can be stupid.

Indeed, getting rid of QA instead of improving it just because people think it's slow is indeed kind of stupid.

> At some point legislators are going to come and ask the question how we stop things like this happening

Bob Martin talks about it a lot, how the software developers of the world need to have an "oath", like the hippocratic oath. Two posts that summarize things well (but there might be more where he talks about these things) https://blog.cleancoder.com/uncle-bob/2011/01/17/software-cr... https://blog.cleancoder.com/uncle-bob/2015/11/18/TheProgramm...

"software developers… need to have an "oath", like the hippocratic oath."

More importantly, the employers of software engineers must have ZERO option to emply a software engineer (anywhere on earth) that doesn't have the same oath.

Doctors have a monospony on their services that makes their oath work: the hospital manager cannot just go hire un-oathed doctors.

Never going to happen in software. Ever.

The practice of medicine was not a licensed endeavor at its inception, and that changed over time. With that in mind, what makes you say "Never going to happen in software. Ever."?

An oath isn’t going to do anything without any way to enforce it legally. The Hippocratic oath is neat but the real teeth are in enforcement against malpractice like civil and criminal lawsuits and a licensing body. You see similar things for lawyers and certain engineers (in commonwealth countries “professional engineer” is a restricted title like Lawyer or MD). Note that just doing that won’t solve all problems either. These licensing bodies regularly publish enforcement actions, so malfeasance continues. Nominally they can help whistleblowers but, as with all regulatory bodies, there’s always a risk of regulatory capture making such actions still peril-filled.

Moreover it’s not even clear this particular work even fall under traditional definitions that would required a licensed engineer as those deal with public safety (bridge construction, buildings, etc) and something like this doesn’t really. We’d need an updated definition that takes into account the software needs of the world (privacy and security, etc).

Doctors don't follow the hippocratic oath in practice, it just isn't a real consideration. If they did none of these covid long haulers or ME patients would have been tortured into worse conditions, nor would all those mentally ill patients have been locked up. Medicine treats the oath like software developers treat most best practices that reduce bug counts, as a nice to have but no one has time for.

> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things.

The phrase "move fast and break things" should be seen as cautionary, not aspirational.

How do you know this culture existed 20 years ago when this system was developed? It is almost certain that a corporate developer in the 1990s would be 100% waterfall.

The issue is very often related to massively complex corporate requirements (the Post Office makes me cringe, even today, with the complexity of their postal system) and then coupled with the ever-present need to keep costs low, especially when designing something so complex.

I doubt anyone building this thought it would be OK to break things!

You are correct. I've seen the Horizon design documents - they are very detailed, thorough and complete. The system was not designed in an Agile manner of iterative design / build first - document later.

Throwing innocent people on jail based on lies (with bonus, corrupt government officials colluding with foreign entities) is the problem here, not software bugs.

Software-based convictions are all about leveraging black box propriety, to hide the flaws that boost conviction numbers.

and

'Justice' is only used ironically now.

Software bugs happen. The trick is to have proper management of the release that takes into account the inevitability of bugs while incentivizing bugs to be identified and fixed without the stakeholders of the project being in a position to have to defend a project as if it is perfect.

I cannot imagine how it must have felt being under the boot of an entire government and it’s corporate partners due to a bug. This is why we are important. A poorly managed IT system with bad incentives puts lives in danger. It is a literal threat to the safety of society. This cannot be stressed enough.

> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives.

I think people see a false dichotomy between making things quickly and making them safe. The fact is in the development of any complex thing, you're going to have bugs, and generally that's okay. But things should be designed to fail safe. Making something that throws errors when something unexpected happens is actually faster and easier than trying (and possibly failing) to handle edge cases; had Fujistu taken that simpler, easier approach then all this pain and suffering would have been avoided.

I still think https://xkcd.com/2030/ has to be taken seriously. You can put a whole load of verification effort into your software, which will undoubtedly make it more reliable. But you are still likely to have some kind of corner case where it breaks down. Software is complex enough for this to be universally true.

The key is how we respond when the software fails. The https://en.wikipedia.org/wiki/Therac-25 case shows an example of what not to do - when hospitals started reporting their machines giving lethal radiation doses to people, the manufacturer doubled down on the computers-are-infallible rhetoric, where they should have put every last effort into investigating. Likewise, the post office should have noticed that a rather excessive number of postmasters were apparently fiddling the books, and investigated. Instead, after it was fairly obvious that the computer was wrong, they pushed the computers-are-infallible line right through the courts, and that is what earned them the "affront to justice" judgment.

The key is how we respond when the software fails.

I agree, but if the first step to solving a problem is understanding that it exists then the first principle here must be to acknowledge that software systems are fallible and therefore any surprising or reasonably contested result they produce should be treated with proper caution until further information can be gathered.

So many of the problems we see when modern technology goes wrong start with assuming it didn't. At that point, it's not even about how you respond to the failure, because you're denying that the failure ever happened. Big software companies with considerable lobbying power seem to be particularly good at convincing people who aren't technical experts, including most politicians, judges, juries and reporters, that this is the case.

A corollary to this is that we desperately need more technological awareness among our politicians, lawyers, journalists and other relevant professions. Tech has become too big to be a minor issue you delegate to some random advisor in a basement office. It affects almost everything we do today, sometimes profoundly, and failing to understand that will inevitably lead to some horrible outcomes as we've seen all too vividly today.

I agree with the sentiment but the example taken for software in this xkcd is wrong. There is a fundamental problem of trust when using software for voting systems that is not linked to the reliability of software but to the nature of voting systems and the properties we want.

Well, there are two problems and which is the fundamental one depends on your prior assumptions.

Some would say it's impossible to build a secure electronic voting system, even if your supplier and their employees were completely trustworthy because between physical tampering, state-level adversaries, the state of the art in software development and the impossibility of proving a negative, such security has never been seen before.

In other words, that it's an unsolvable technical problem.

Others would say it's impossible to build a secure electronic voting system even if we were capable of creating flawless bug-free and tamper-proof software and hardware because the supplier will always be able to introduce undetectable bugs if they want to, and no supplier can ever be perfectly trustworthy.

In other words, that it's an unsolvable social problem.

Yeah, electronic voting is essentially like having a person in the voting booth that you have to tell your vote and trust that they will tally it correctly. [1]

It doesn't matter whether voting machines are actually secure, they probably mostly are right now, but whether a layperson can have faith in the system.

Paper voting is very secure if you involve people from opposing parties in the process and attacks are not very scalable. Most people can think of and understand mitigations for certain kinds of attacks. And if paper voting is too expensive for your country, you have bigger issues. [2]

[1] https://www.youtube.com/watch?v=LkH2r-sNjQs

[2] That said, I don't see how secure electronic voting can possibly be cheaper than paper voting. For voting machines to be secure, you have to manufacture them in a very audited manner, with little to no foreign sourcing of parts, you can't leave the machines unattended for long periods of time (aka, reusing them between elections is probably a no-go) and you have to build them in manner that is secure against voters tampering with them in their private booth.

Something about "pride goeth before a fall" and "the one thing you can expect a manager to do is whatever shields them from liability".

This won't change until executives go to jail.

A few years ago, we were fighting against tight deadline and skipping unit tests, QA, processes, etc. Someone brought up one of the recent major breach (Equifax?). Developers started to say that people will go to jail. Basically, devs were using this breach to imply that they will personal responsibility for releasing a product that might have security flaws. Our director laughed and said no one will go to jail and if our product ever got in trouble, they will personally take responsibility.

> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things.

This is extremely domain dependent, and should be handled as such. And in some cases it already is - look at the testing / verification space shuttle code goes through vs your friends cat video side project website.

I disagree. The needed quality of your software changes dramatically depending on what it is used for. Something that helps someome semi-automate their workflow where bugs just mean they have to do things manually (as things that are wrong fail quickly and obviously) is something where what matters is speed of release and features. Something calculating how long people should be in prison needs to care about quality.

Anyway, the courts are the ones that have the biggest share of the blame here. Believing an unreliable witness for the prosecution is a common cause of injustice.

> At some point legislators are going to come and ask the question how we stop things like this happening

If the problem is the software, then use less software. Perhaps we shouldn't simply take it as a given that moving processes into software isn't always the right move?

Reply on news.ycombinator.com