Hacker News Reader

Comment by MadBohr | Hacker News Reader

I disagree, you're just passing the buck. Accountability needs to be had at all levels. If an engineer writes a bug into code like this (deliberate or not) and such a bug results in somebody's death, the engineer should be held accountable just as much as the person who approved its release. The executive could just as easily say "my engineers promised me it was fully tested", etc. Engineers could say "yep it was, but that was an edge case we missed" or something like that. In any case, there needs to be shared accountability. Maybe execs take the brunt, but engineers should not be allowed to write code that kills people (inadvertently or otherwise) and face zero consequences.

View on HN

What software developer would ever sign on to a project where they could be held criminally liable for a single bug?

Do you want software development to turn into healthcare, where every developer needs millions of dollars of malpractice insurance? Because shit like this will turn it into a healthcare like system real quick.

I'm sure plenty of people would but that isn't the point. If you're writing code that potentially costs people their lives, you need to be able to be held accountable otherwise it will lead to negligence. This isn't a new problem... maybe for the software space, but not for industry as a whole.

Humans don't suddenly become perfect actors just because incentives align. The stress of that risk and efforts taken to mitigate it seems like it would actually make the software worse.

It's up to the product (the collective of individuals that deliver the product) to address and mitigate the risk it creates, that's not solely on the shoulders of individual software contributors.

If A writes a generic computer vision algorithm and open sources it, B integrates that into a "is this a bomb or not" product with a white paper outlining its failure rate in a specific situation, then C sells that product to D who uses it in an entirely different situation and E gets blown up... who gets sued? It definitely should be somebody, there should certainly be a liability and incentive to avoid such a liability but I it probably lies somewhere in C-D space, not A-B space.

If the only options you’re presenting are “move fast and break things” where those things are human lives, or introducing burdensome bureaucracy, I’ll take the bureaucracy. Time and again society chose that latter option, and it will again. Unaccountablility is worse than regulation, and history has shown that repeatedly.

This is quite the strawman is it not? I said nothing about "move fast and break things."

What software developer would ever sign on to a project where they could be held criminally liable for a single bug? Do you want software development to turn into healthcare, where every developer needs millions of dollars of malpractice insurance? Because shit like this will turn it into a healthcare like system real quick.

How else to interpreted that? When a single bug can cause loss of life, and given that this in a thread about Uber, it’s hard to draw other conclusions. By all means though, offer another perspective on how regulating industries with significant number of lives on the line can’t manage regulation. While you’re doing that, I’d point to the aerospace sector which seems capable of both innovation and regulation.

There's a difference between holding someone criminally responsible for a bug in code that they wrote, and some sort of regulation. They are not the same.

For example: https://en.wikipedia.org/wiki/Boeing_737_rudder_issues

Bad example for two reasons. First:

Although the NTSB investigated the accident, it was unable to conclusively identify the cause of the crash. The rudder PCU from Flight 585 was severely damaged, which prevented operational testing of the PCU.[3]:47 A review of the flight crew's history determined that Flight 585's captain strictly adhered to operating procedures and had a conservative approach to flying.[3]:47 A first officer who had previously flown with Flight 585's captain reported that the captain had indicated to him while landing in turbulent weather that the captain had no problem with declaring a go-around if the landing appeared unsafe.[3]:48 The first officer was considered to be "very competent" by the captain on previous trips they had flown together.[3]:48 The weather data available to the NTSB indicated that Flight 585 might have encountered a horizontal axis wind vortex that could have caused the aircraft to roll over, but this could not be shown conclusively to have happened or to have caused the rollover.[3]:48–49

On December 8, 1992, the NTSB published a report which identified what the NTSB believed at the time to be the two most likely causes of the accident. The first possibility was that the airplane's directional control system had malfunctioned and caused the rudder to move in a manner which caused the accident. The second possibility was a weather disturbance that caused a sudden rudder movement or loss of control. The Board determined that it lacked sufficient evidence to conclude either theory as the probable cause of the accident.[2]:ix[3]:49 This was only the fourth time in the NTSB's history that it had closed an investigation and published a final aircraft accident report where the probable cause was undetermined.[4]

Second:

In 2004, following an independent investigation of the recovered PCU/dual-servo unit, a Los Angeles jury, which was not allowed to hear or consider the NTSB's conclusions about the accident, ruled that the 737's rudder was the cause of the crash, and ordered Parker Hannifin, a rudder component manufacturer, to pay US$44 million to the plaintiff families.[16] Parker Hannifin subsequently appealed the verdict, which resulted in an out-of-court settlement for an undisclosed amount.

You interpret it as written, which is that holding developers routinely criminally liable for bugs is going to have very negative effects. One of them is that the only developers you'll get are precisely those too unwise to realize what an incredibly stupid deal that is, no matter what the pay rate is. I don't think I'd like to see all my critical software written by such "unwise developers".

I have no problem "piercing the veil" for egregious issues. I'd have no problem holding a developer liable for failing to secure a project but just continuing on rather than quit. But "Let's just hold all the engineers criminally liable all the time!" is a bad idea and it is not already done for a reason.

It’s not done because software development is an unregulated shitshow full of wildly unethical companies scrambling for the bottom. It’s not unlike early aerospace, or early medicine, or any frontier which develops rapidly before legal frameworks inevitably close in.

It's also not done because it is mathematically impossible to certify software as 'bug-free' in the general case.

Software isn't like civil engineering where you can mathematically prove that a design is sound.

This is not true at all. First of all, there's no such thing as being able to mathematically prove a design is sound in any engineering discipline, software or non-software. After all, it is infeasible if not impossible to encapsulate all the details of the implementation of _any_ system in mathematics or any other system of reasoning (down to every last atom, if you stretch your imagination).

All we have in engineering (non-software) is something like safety factors and confidence, and this is done with (usually) rigorous mathematical models as well as loads and loads of testing to fill in the gaps of mathematics (think unknown constant/parameters, assumptions, etc).

None of this is impossible to do for software. There are systems that enable one to do easy/entry level verification (such as something like TLA+), to much more complicated reasoning (something like COQ). This will allow the system designers to gain confidence in if the system will work and gain understanding about under what scenario they will fail. Contrast this with the existing software landscape, which is mostly, at least from my perspective, just let me write some stuff until things do approximately what I want. Even at the top of the ladder, I feel the tests conducted are "adhoc" at best and with none of the rigours that you associate with traditional engineering fields.

Healthcare costs are not unreasonable in much of the developed and developing world. Most countries have better outcomes and lower costs than here in the US. As another commentor says, healthcare seems to be doing fine; you seem to be assuming the US is the norm when it isn’t.

> Healthcare costs are not unreasonable in much of the developed and developing world. Most countries have better outcomes and lower costs than here in the US. As another commentor says, healthcare seems to be doing fine; you seem to be assuming the US is the norm when it isn’t.

I'm going to cauterize the off-topic debate about the US healthcare system by pointing out that OP was talking about the expense to doctors of malpractice insurance, not about costs to the patients or medical outcomes.

Malpratice liability varies widely by country, but it's a non-trivial expense for doctors everywhere, and significantly higher in states with strong tort liability for doctors.

It's hard to imagine a world with criminal liability (or tort liability) for software engineers that doesn't ultimately end up with a system of insurance for engineers, roughly analogous to the medical malpractice insurance system for physicians.

I am afraid that you are introducing the off-topic debate. The end goal of healthcare is better outcomes for lower prices. Likewise, the end goal of engineering should be better technology for lower costs.

That healthcare in other countries is able to achieve this in spite of the medical malpractice insurance system points to the fact that such a system is not certain have to have the deleterious effects you confidently assume.

Whether it is a burden for engineers is another question. But the article and the discussion aren’t about the inconveniences faced by the engineers who programmed this system.

>system of insurance for engineers

Which, as someone else noted, exists and is probably a good idea if you're an independent consultant or possibly a professional (i.e. licensed) engineer who signs off on drawings or other documents for clients or regulators.

Criminal liability is a different situation as there are very few industries with specific criminal liabilities (finance maybe).

But there are many industries where civil liabilities are required. In fact, any software independent consultant is civilly liable for their work, but it’s not specific to software.

IEEE has a section in their member toolkit that goes into why professional liability insurance is needed, https://m.ieee.org/membership_services/membership/discounts/...

The costs aren’t that high or at least they weren’t 15 years ago when I purchased it for less than $1k/year for $1M in coverage. Most people need this even if they think they are safe. If you’re the one who wrote the deployment script that erased $1M in data, it won’t be entirely mitigated that the script made it through qa.

Also interesting is that the engineer who wrote the Uber software is currently liable for criminal negligence, like pretty much everyone else. But you would have to prove culpability. I can’t find any examples of software engineers convicted so it’s hard to tell who goes to jail-developer, qa, or executive.

More info on criminal/civil negligence- https://www.theblanchlawfirm.com/?practice-areas=criminal-ne...

Nobody in their right mind would work with such liability without insurance, which is all well and good for civil liability, but insurance won't help if you're going to jail.

I think we may be arguing different things.

Almost all employees have the possibility of criminal negligence based on their work. For programmers, this could mean that if you fuck up the code for a pacemaker and someone dies, you could go to jail. That’s a big risk and I can’t find any programmer who has been found culpable for someone’s death. This is the current law in the US.

If Uber was negligent in its code, then the programmers could go to jail. They have programmers and they work and assume this extremely low risk.

Now maybe you’re arguing that some special law should or should not exist for Uber drivers.

This happens all the time in aerospace. You need to sign off the software personally and you need to be an accredited engineer to be allowed to do that.

Reply on news.ycombinator.com