Hacker News Reader

Comment by dimitrios1 | Hacker News Reader

We know exactly how to fix it. Our cowardly politicians and toothless regulatory agencies are not up for the challenge.

For every piece of software that can directly and materially harm someone's life like this, there should be a chain of responsibility. And within that chain, there should be legal recourse and, in most cases, penal consequences, especially in the case of inadequate software quality/testing/validation, should the software fail to perform its task correctly. Bonus side effect, software quality will go up across the board in the industry.

View on HN

It will never be the case that software will be perfect. We can get closer and closer, but the closer we are the more expensive the next step in closing the gap is.

While I do agree that making software better/more reliable is a good goal, I believe we would be better off making the system as a whole more robust; the system that includes humans. For every situation where a piece of software has control of something that effects society (individual, group, etc), there should always be a clear and direct means of appealing / pushing back on the decision that was made. Those means should involve a human reviewing the information and making a decision based on that information, not on what the computer said. There's thread after thread of us saying the exact same thing about companies like Google and Facebook; it should apply as a general rule.

No one is arguing that software must be perfect. But we aren't really even trying. Most software is written in extremely error prone languages without adequate testing.

You don't hear anyone saying we should throw out finite-element analyses and other computational verification methods when designing bridges because bridges can never be perfectly secure. Yet that is exactly the sentiment I often hear on software.

Are bridges and software comparable in complexity though? How many engineers work on designing a bridge compared to software developers that are writing a program? There are more than an order of magnitude more software developers in the US than civil engineers.

Now think about the state of US infrastructure. Does it inspire confidence for the future?

I'd say a lot of software is comparable in complexity to the large bridges that exist. Of course there are massive software projects that dwarf any bridge ever build. But a lot of software is only moderately complex.

I don't know about infrastructure in the US. I don't live there. I'm happy with the infrastructure in west Europe though. I wish that much care was put in to the software I use every day.

There is no such thing as perfect software when requirements change with time. We need adaptable software instead.

Like all our governance mechanisms have a built-in system of constant change.

I agree with other commenters. And I think another part of the problem is the consumer model of software. In a pen and paper system if there was some reason why a record was special or different from the others, you could just attach a note to it and the next person who picked up the record could read your note. Custom software systems deny that sort of ad-hoc flexibility from the people using them. There’s no way to do anything that wasn’t planned for and programmed in. So office workers who use flows managed by custom software are actively disempowered from authorship over their own workflows.

That’s one of the reasons I think Excel (and tools like Notion) are so popular - the people on the ground can learn to express themselves in the full context of the tools. I think this sort of software is far more important than we give it credit for. (It’s an invisible problem to us, because we can change the software.)

It’s not the software makers who are committing the crimes. It’s the people abdicating responsibility to software. You can’t wipe your hands of releasing a prisoner on schedule by delegation that to software. The software can help you with your task, but if it’s brought to your attention that there’s a mistake, your failure to promptly fix it is on you.

It's both. Look at the amount of unnecessary waste created by the abysmal Android update policy.

Most of the time, liability for defective software should be civil liability in tort or contract. In most cases where something bad has occurred involving software, it’s going to be hard for the developers of the software to have the sort of mental state and unattenuated causal connection with the occurred harm that we typically require for criminal liability for reckless conduct.

I’m not saying it can’t happen, but it would be very unusual circumstances, especially since there’s usually an operator of the software sitting between the developer and the person harmed by the software.

Which abysmal android update policy? What waste?

Imho that wouldn‘t cause quality to go up. It would just make it more expensive to develop and to fix bugs. Even more cover your ass would go on.

Or at least a huge share of that burden needs to be on the client so that they define and then test and control the SW they receive properly.

The problems with the software sound like typical big software project problems. Trying to cover a huge breadth of use cases with lots of very important tiny details and released in a big bang (one migration). It sounds like more of a project mgmt problem than a software problem to me.

But maybe I am just a hammer and see nails everywhere.

My cynical take is that the lack of accountability is exactly what makes software enabled bureaucracy so appealing. If this true, there is no incentive to change.

It's a cost/benefit analysis. They'll be taken to court by a few inmates, have to pay them or settle out of court and bill the government for their stays. Since everything is a matter of money, what the company will do is what brings them the most profit after they paid their legal bills.

It seems keeping inmates longer pays better than releasing them on time.

This is part of it. Otherwise it’s just way cheaper to run a faceless bureaucracy that sometimes throws people under the bus when a mistake has been made.

Or things don't get much better, and management finds a way to make it someone else's problem. Do you think it's the CEO or the people making these decisions that will be locked up? We'll just be arresting whatever yes men show up to be the pawn in the stupid game someone else architected.

More people working with a gun to their head. I'd rather the gun be pointed at the person who already has a gun pointed at me, instead of both barrels facing in my direction.

Yeah but the cost of that chain will also rise.

If I'm (or my company is) personally on the hook for bugs, then I'm going to adopt a NASA-like software quality regimen, pushing up the cost of the product.

Every single part of the software stack below me, from hardware, OS, compiler toolchain, disavows responsibility so if I have to absorb all the risk, the product is going to be mind bogglingly expensive.

that doesn't sound bad to me if we are dealing with people's life or people's freedoms

I have to say that sounds exactly how this kind of software should be built.

We're not talking about the newest social media hype. This software actually matters. Specially since today most of these bureaucratic processes can't be done without these softwares.

>I have to say that sounds exactly how this kind of software should be built.

You see this in every topic.

Every "muh pride in muh trade" person says something like this about the relevant trade but the fact of the matter is that the world runs on off-brand duct tape, harbor freight tools, walmart jeans, economy tires, and all sorts of other "value" solutions and the race to the bottom is what has given us much of the modern world that we take for granted.

A balance needs to be struck. And it generally needs to be struck further toward the "quickly and cheaply build it like crap but make it easy to override or reset" portion of the available solution space than anyone pontificating about quality on the internet will readily admit.

We don't say "perfection is impossible" when it comes to bridges collapsing. We understand that yes, on rare occasion a bridge WILL collapse, but we go and find the people responsible, and we still hold them accountable.

This is a level of accountability that basically every other field of engineering is held to, and they've all risen to the challenge and left the "off-brand duct tape" behind.

Even within programming, planes don't fall out of the sky daily, so I feel safe assume the aerospace programmers are comfortable working with a high degree of responsibility. High speed traders are dealing with million-dollar stakes and a single mistake can make the news. I'd expect they've got a very accountable culture where people get fired when that happens.

There are costs, yes, but there's also costs to keep 733 people illegally imprisoned - we're talking two man-years of peoples lives lost every DAY this goes on.

Why not hold the people accountable that deployed the tool? Ultimately the tool helps a human do the job. It doesn't do anything on its own. If a contractor shows up to do repairs in your house, but their hammer is faulty and the head flies through your window you don't talk to the hammer manufacturer. You talk to the contractor. It's the contractor's job to deal with wherever they got the hammer from.

If software developers are held responsible for the software then expect costs to multiply. Nobody would directly sell you software either - they'd sell you a hardware and software bundle that you must use exactly as the developers say. If you input a value that's out of bounds then that's on you. The software also won't get updates and it will run on 20 year old hardware. That's not too dissimilar to what we have in aerospace, right? And developers aren't even held responsible there! It's the companies, so expect it to be worse than even that.

Would you blame the people who paid for the bridge for the collapse? Should they have understood the details and flagged where corners were cut.

When it comes to critical system I think it's fair to say that the engineers who build it are the only ones who can fully understand the risk.

This is the point behind accreditation. It forces the supplier to maintain a minimum bar for services to protect the reputation of the industry.

In real life the engineers don't police themselves.

Before a bridge, house or even patio deck with a foundation is used a safety inspector needs to give approval.

Are safety inspectors intended to validate the design of a bridge? Or just that construction and materials are up to spec?

>We don't say "perfection is impossible" when it comes to bridges collapsing.

Yes we do. People on the internet might not but look at the formal documentation that goes with any bridge plans. It will talk about factors of safety, various loads, environmental conditions and establish a set of constraints outside of which the bridge is not expected to perform as advertised.

>speed traders are dealing with million-dollar stakes and a single mistake can make the news

It's really easy to put HFT a pedestal when you can't inspect it up close but I assure you that for every Citadel and P72 there is half a dozen firms with sloppy software that goes absolutely crazy if non-ideal but foreseeable things happen. These people are making money hand over fist (kind of) by building to the minimum. There's one firm I want to name because of how much everything they have is held together with duct tape but they're nice guys so I won't.

> the world runs on off-brand duct tape

So very, very true.

It is already costing as it should have that level of quality - a 24 million dollar system should be hard to justify that cost.

The other option is to accept we have mediocre software that creates a number of problems that we are willing to accept; NO, not when people's lives are at stake.

Good, this is precisely the objective.

If there’s penal consequences for bad software, you can bet that the development cost will easily 10x overnight.

Is there a better option? This is software that has to 100% work in order to be trusted. If the software cannot be trusted, additional process has to fill in to verify the decisions the software is making. There's no way to deliver a good solution cheaply.

All software has bugs. This is the mantra we are taught and for a good reason.

The answer is human oversight.

I wouldn’t work on such software.

You would for 10x the salary.

> Our cowardly politicians and toothless regulatory agencies are not up for the challenge.

Because their constituents want people to be punished and if the inmates have to suffer a little extra so be it, "they shouldn't have committed a crime."

Our society is severely lacking in empathy.

> We know exactly how to fix it. Our cowardly politicians and toothless regulatory agencies are not up for the challenge. For every piece of software that can directly and materially harm someone's life like this, there should be a chain of responsibility.

No, you know how to blame people and punish people, but that doesn't mean you know how to deliver custom bespoke software for a price that the various government agencies can afford which doesn't have bugs that severely hurt peoples lives.

In fact, punishing people is not going to accomplish that.

That's the problem with a legislature that thinks it can pass any law it wants - let's take into account this new variable X that our software has no way of collecting or measuring - without looking at the feasibility of actually implementing the law given the infrastructure available, and without approving a corresponding budget for software upgrades to actually enact the law, and taking into account how much time it would take to write, test, deploy, and then train people to use the new software instead of just issuing streams of mandates like Emperor Norton and expecting the mandates to materialize into existence like the morning dew. And if said morning dew does not appear, then we can punish and sue the people in charge when they tell us there is no way they can do what we are asking them.

Of course there is blame on the prison leadership for covering things up and that leadership should be fired, but you can punish and sue people all day long and it's not going to result in any good code being written. Punish enough people, and it will just result in the Law being repealed.

The problem with this type of bespoke code is that it has exactly 1 customer, so it's going to be horrendously expensive while also being buggy and quickly thrown together compared to software whose development costs are leveraged over millions of customers. And then what happens next year when some crusader decides that they need to take some other new variable into account? Constantly changing requirements, underspecified projects, one-off projects whose schedules are impossible to estimate, and cash strapped local governments. Yeah, that's a recipe for success.

This is why everyone hates enterprise software, but even enterprise software has tens of thousands of customers. Bespoke software for the Arizona prison system -- forget it.

> And within that chain, there should be legal > recourse and, in most cases, penal consequences,

Wikipedia says, "Under common law, false imprisonment is both a crime and a tort".

There is a chain. There is legal recourse. And there are considerations in government IT that you would not believe and they are incredibly difficult to deal with on minimal resources. It has to be harder than the private sector and this application isn't any different than buggy mainframe software run by major banks. It sits and gets crufty.

This is a new system that replaced a previous one not that long ago. This isn't some crazy old thing running COBAL on a VAX somewhere that nobody understands anymore.

The old COBOL crap is more likely to have been implemented by someone with a clue.

The “new” systems are usually aping the old system behavior. In one case, I ran into a system where some company converted COBOL transactions into Java with some sort of automated tool to put the legacy system “on the internet”.

I have no idea what that means, for newer is not necessarily more supportable. Who knows what the system is - maybe they had a multi-million dollar SAP implementation to manage prisons, and now you’re looking for functional support of that platform after it was customized all to hell, you need a-team SAP resources, not the new kid at Wipro... I can only imagine what’s behind that curtain. It’s the government so I imagine the worst.

Maybe every contract like this should be programmed to the same API twice. Then you could at least compare it the two pieces of software agree. You check the disagrees and get the companies to fix them (should be part of the contract).

And don't tell me you can't buy two CRUD applications for 24 million dollars. It's a silly amount of money for such a buggy application.

> cowardly politicians

They aren't cowardly; they are responding rationally to a constituency that hates "criminals". Prioritizing fixing discriminatory systems (such as this software, or "stop and frisk", or the death penalty) is bad electoral politics for "tough on crime" politicians.

The politicians and toothlessness of the regulatory agencies is a direct result of the electorate. The electorate likes these sorts of outcomes, and any politician that goes against them will find themselves either primaried or drummed out of office.

Nobody will take that contract. Any mistake in the government specifying what they want you to build can be politically manipulated into a legal matter with some probability you'll end up in prison.

Not at all: any mistake in the government specifications is the government's problem. If you have requirements and a requirement tracing matrix to deliverables and you deliver what is required, you are good.

But usually in most parts of the world these government contracts are intentionally made ambiguous for so many reasons including corruption and incompetence of the people writing requirements.

At what point do you feel the developers - the ones who actually wrote the code - should be held legally responsible for that code's execution?

This headlining issue is a specification change and it is an administrative rather than an engineering failure to knowingly rely on outdated software. The article also refers to other software problems but in scenarios like this the people who write the code (as well as the people who operate it, such as corrections department IT staff) tend to have no decision making power whatsoever:

>“It was Thanksgiving weekend,” one source recalled. “We were killing ourselves working on it, but every person associated with the software rollout begged (Deputy Director) Profiri not to go live.”

Reply on news.ycombinator.com