It's unfortunate, but unsurprising, the first pedestrian killed by an experimental self-driving car allowed on public roads was in Arizona.
We need to discuss how the developers self-driving cars will be held accountable for the crimes they commit. There is no reason the person who programs or the person who makes money from a self-driving car should be held less accountable for a crime that if committed directly by a person would almost certainly result in jail time. You can’t replace yourself with a computer program and then choose to take only the benefits and not the responsibilities.
Moreover it is not clear to me that not holding the companies that create industrial robots that kill people criminally responsible is what most people would consider just. Again, I think it’s just that there is a massive difference in the scale of exposure; there were not enough interested people to have a debate.
We have held humans responsible because assuming a correctly functioning car they are performing the most complex and risky task, and are most able to cause problems. Likewise self-driving car software performs a complex and risky task in which failure can have serious consequences.
As a pedestrian you already run a significant risk of being killed by a car. To the extent that we hold autonomous car makers responsible for these deaths (and I'm not saying we shouldn't), we should hold non-autonomous car makers responsible for the deaths their vehicles cause as well.
Slow your roll. Nobody know why this person died yet.
In this case, the operator was an AI that was negligent and it was unlicensed/unregulated. That's a new scenario. In the human case a person might go to jail for negligent vehicular manslaughter. What does 2 years of jail time look like to an AI? What does a suspended license look like to an unlicensed entity?
For scale: the risk of death from a self driving car will probably be less than the current risk of death from normal cars, and will definitely be less than the risks incurred in the 20th century from cars, buildings, etc.
Self-driving cars are definitely a new and large legal development, but there's no reason to think existing legal principles can't handle them.
As the GP states, the entire reason Uber is testing in Arizona is because their state government completely got rid of reporting regulations which were present in CA; the status quo is decidedly not the same as it is for established technologies.
As for scale, look at the other comments where people analyze the risk posed by self driving cars. Your assumption that the risk of death from self-driving cars is less is not backed up by the evidence.
It’s fine to say that self-driving cars might eventually be better drivers than humans, just like robots might eventually be better at conversing than humans.
There is no reason self-driving cars can’t be be tested in private. Uber can hire pedestrians to interact with them—I don’t volunteer to be their test subject by deciding to take a walk.
Self-driving cars are a new and important industrial development that will require adjustments to policy. They don't require revolutionary new legal principles.
The executives should be held accountable, not the developers.
Do you want software development to turn into healthcare, where every developer needs millions of dollars of malpractice insurance? Because shit like this will turn it into a healthcare like system real quick.
But there are many industries where civil liabilities are required. In fact, any software independent consultant is civilly liable for their work, but it’s not specific to software.
IEEE has a section in their member toolkit that goes into why professional liability insurance is needed, https://m.ieee.org/membership_services/membership/discounts/...
The costs aren’t that high or at least they weren’t 15 years ago when I purchased it for less than $1k/year for $1M in coverage. Most people need this even if they think they are safe. If you’re the one who wrote the deployment script that erased $1M in data, it won’t be entirely mitigated that the script made it through qa.
Also interesting is that the engineer who wrote the Uber software is currently liable for criminal negligence, like pretty much everyone else. But you would have to prove culpability. I can’t find any examples of software engineers convicted so it’s hard to tell who goes to jail-developer, qa, or executive.
More info on criminal/civil negligence- https://www.theblanchlawfirm.com/?practice-areas=criminal-ne...
Almost all employees have the possibility of criminal negligence based on their work. For programmers, this could mean that if you fuck up the code for a pacemaker and someone dies, you could go to jail. That’s a big risk and I can’t find any programmer who has been found culpable for someone’s death. This is the current law in the US.
If Uber was negligent in its code, then the programmers could go to jail. They have programmers and they work and assume this extremely low risk.
Now maybe you’re arguing that some special law should or should not exist for Uber drivers.
How else to interpreted that? When a single bug can cause loss of life, and given that this in a thread about Uber, it’s hard to draw other conclusions. By all means though, offer another perspective on how regulating industries with significant number of lives on the line can’t manage regulation. While you’re doing that, I’d point to the aerospace sector which seems capable of both innovation and regulation.
For example: https://en.wikipedia.org/wiki/Boeing_737_rudder_issues
Although the NTSB investigated the accident, it was unable to conclusively identify the cause of the crash. The rudder PCU from Flight 585 was severely damaged, which prevented operational testing of the PCU.[3]:47 A review of the flight crew's history determined that Flight 585's captain strictly adhered to operating procedures and had a conservative approach to flying.[3]:47 A first officer who had previously flown with Flight 585's captain reported that the captain had indicated to him while landing in turbulent weather that the captain had no problem with declaring a go-around if the landing appeared unsafe.[3]:48 The first officer was considered to be "very competent" by the captain on previous trips they had flown together.[3]:48 The weather data available to the NTSB indicated that Flight 585 might have encountered a horizontal axis wind vortex that could have caused the aircraft to roll over, but this could not be shown conclusively to have happened or to have caused the rollover.[3]:48–49
On December 8, 1992, the NTSB published a report which identified what the NTSB believed at the time to be the two most likely causes of the accident. The first possibility was that the airplane's directional control system had malfunctioned and caused the rudder to move in a manner which caused the accident. The second possibility was a weather disturbance that caused a sudden rudder movement or loss of control. The Board determined that it lacked sufficient evidence to conclude either theory as the probable cause of the accident.[2]:ix[3]:49 This was only the fourth time in the NTSB's history that it had closed an investigation and published a final aircraft accident report where the probable cause was undetermined.[4]
Second:
In 2004, following an independent investigation of the recovered PCU/dual-servo unit, a Los Angeles jury, which was not allowed to hear or consider the NTSB's conclusions about the accident, ruled that the 737's rudder was the cause of the crash, and ordered Parker Hannifin, a rudder component manufacturer, to pay US$44 million to the plaintiff families.[16] Parker Hannifin subsequently appealed the verdict, which resulted in an out-of-court settlement for an undisclosed amount.
I have no problem "piercing the veil" for egregious issues. I'd have no problem holding a developer liable for failing to secure a project but just continuing on rather than quit. But "Let's just hold all the engineers criminally liable all the time!" is a bad idea and it is not already done for a reason.
Software isn't like civil engineering where you can mathematically prove that a design is sound.
All we have in engineering (non-software) is something like safety factors and confidence, and this is done with (usually) rigorous mathematical models as well as loads and loads of testing to fill in the gaps of mathematics (think unknown constant/parameters, assumptions, etc).
None of this is impossible to do for software. There are systems that enable one to do easy/entry level verification (such as something like TLA+), to much more complicated reasoning (something like COQ). This will allow the system designers to gain confidence in if the system will work and gain understanding about under what scenario they will fail. Contrast this with the existing software landscape, which is mostly, at least from my perspective, just let me write some stuff until things do approximately what I want. Even at the top of the ladder, I feel the tests conducted are "adhoc" at best and with none of the rigours that you associate with traditional engineering fields.
I'm going to cauterize the off-topic debate about the US healthcare system by pointing out that OP was talking about the expense to doctors of malpractice insurance, not about costs to the patients or medical outcomes.
Malpratice liability varies widely by country, but it's a non-trivial expense for doctors everywhere, and significantly higher in states with strong tort liability for doctors.
It's hard to imagine a world with criminal liability (or tort liability) for software engineers that doesn't ultimately end up with a system of insurance for engineers, roughly analogous to the medical malpractice insurance system for physicians.
That healthcare in other countries is able to achieve this in spite of the medical malpractice insurance system points to the fact that such a system is not certain have to have the deleterious effects you confidently assume.
Whether it is a burden for engineers is another question. But the article and the discussion aren’t about the inconveniences faced by the engineers who programmed this system.
Which, as someone else noted, exists and is probably a good idea if you're an independent consultant or possibly a professional (i.e. licensed) engineer who signs off on drawings or other documents for clients or regulators.
It's up to the product (the collective of individuals that deliver the product) to address and mitigate the risk it creates, that's not solely on the shoulders of individual software contributors.
If A writes a generic computer vision algorithm and open sources it, B integrates that into a "is this a bomb or not" product with a white paper outlining its failure rate in a specific situation, then C sells that product to D who uses it in an entirely different situation and E gets blown up... who gets sued? It definitely should be somebody, there should certainly be a liability and incentive to avoid such a liability but I it probably lies somewhere in C-D space, not A-B space.
The person who profits the most should be held the most responsible. But the separation of roles between the executives and the developers is likely to mean that no one gets punished at all.
Look, I'm all for developers and (software companies in general) to be considering the ethical implications of the work they do, and the moral obligations that they take on as a result of it. However:
> We need to discuss how the developers self-driving cars will be held accountable for the crimes they commit. There is no reason the person who programs or the person who makes money from a self-driving car should be held less accountable for a crime that if committed directly by a person would almost certainly result in jail time. You can’t replace yourself with a computer program and then choose to take only the benefits and not the responsibilities.
This is a bad mentality to take with postmortems for software failures in general, at least from the outset. You need to look at the underlying factors that contributed to the issue, not simply looking for a person to assign blame to. It's possible that negligence is the underlying cause, but not necessarily - and even if negligence is a cause, what were the other cultural factors that led to the negligence happening, without being caught somewhere else in the pipeline? It's tempting to look to assign blame, but if you do that, you'll actually miss out on the systemic improvements that would be necessary to prevent similar incidents in the future.
But moreover, this is a bad outlook to take here, because this wouldn't be criminal behavior if committed by a human. From the best we can tell, given the details available so far, it's an accident, and it's very rare for criminal charges to even be considered in accidents like these, unless it's a hit-and-run.
I can assure you that a human who is driving carelessly would be held criminal liable. Why do you assume an accident that was severe enough to have resulted in a persons death—the car didn’t just scrape them because they ran across the street—is not due to a reckless programming?
Do you see evidence that the car was "driving carelessly"? That's an honest question - from the reporting so far, it doesn't seem clear what the underlying cause was.
Secondly, this is demonstrably false: most pedestrian fatalities by vehicles do not result in criminal charges. If you don't believe me, look up the stats. Or talk to the countless bikers' advocacy groups that have been lodging this exact complaint for decades: drivers are not generally held criminally responsible, unless there are mitigating circumstances (the driver is drunk, the accident was a hit-and-run, etc.).
> Why do you assume an accident that was severe enough to have resulted in a persons death—the car didn’t just scrape them because they ran across the street—is not due to a reckless programming?
When a pedestrian dies, just because they died, that doesn't mean the driver is automatically responsible. It could have been the pedestrian's fault, or it could have been the driver's fault. Or it could be both. Or it could even be neither (a true accident, with no assignment of blame).
The same thing holds here. You can't assume that this is the result of "reckless programming", and to be entirely blunt, by jumping to that conclusion on the basis of literally no evidence whatsoever (and misinterpreting existing case law on vehicular accidents in the process), you're actually undermining the success of any future efforts to prevent these sorts of accidents in the future, whether or not it ultimately turns out to be the fault of someone at Uber.
I accept your point that this accident might not have led to criminal charges if a human had been responsible. But I don’t waver on my argument that if a human driver would have been held criminally responsible for this accident, then we should we hold the executives (or in extreme cases programmers) of Uber responsible in exactly the same way, whether that be criminal or not.
Finally, with humans and pedestrian fatalities many cases involve drunk driving or sleepy driving. Self-driving cars can’t get drunk or sleepy; they can just have bad programming or bad hardware, both installed by their manufacturer.
What should be passed (but I can't see how) is a percentage of allowed deaths, at least in the early years, and set it to something like 5-10% of the current rate, reducing downwards to 1% after 20 years.
People will die from self driving cars, and undoubtably their will eventually be a case that is 100% the self driving car's fault. The benefit of self driving cars comes from the mistake being permanently fixed, while with human drivers it can be committed over and over again.
There needs to be some kind of protection on the companies (and obviously the developers, I've never heard someone try to say they should be held responsible before) from lawsuits. Otherwise all it'll take a is a small handful beefore companies will just let it die.
Civil engineering and medical device manufacturing seems to be doing fine, despite having similar principles of engineers' liability.
The idea that software engineers should be held responsible for something that (as far as we can tell so far) was an accident and not the result of negligence or malice is several orders of magnitude beyond the level of liability that civil engineers and medical device manufacturers have.
Nobody said that. The original comment said developers should be held "accountable for a crime that if committed directly by a person would almost certainly result in jail time" [1].
The standards from medical devices and/or civil engineering, with the associated licensing requirements and verification processes, make sense. Even in the case of a careless mistake or strategic oversight, individuals who could have known but nevertheless signed off should be identified, if not explicitly punished.
> Nobody said that.
Well, they quite literally did, because the original comment in this thread was:
> We need to discuss how the developers self-driving cars will be held accountable for the crimes they commit. There is no reason the person who programs or the person who makes money from a self-driving car should be held less accountable for a crime that if committed directly by a person would almost certainly result in jail time. You can’t replace yourself with a computer program and then choose to take only the benefits and not the responsibilities.
I guess you can quibble about the difference between "accountable" and "liable", but that's not a discussion that's particularly interesting to have here, especially given OP's other comments in this thread which make it quite clear that this is what they had in mind.
If the developer was negligent or malicious in their duties, why not prosecute them?
If we are going to be arguing from a utilitarian standpoint, suppose we hold the executives of self-driving car companies as responsible as if they were themselves drivers. Then if self-driving cars truly are safer as you optimistically claim, both fewer people will die from accidents involving them and fewer people will go to jail for those same accidents. Seems like a win to me.
Whith so few self driving cars that number sold be zero. If you can't assure safety with a few cars whith a human as backup, you should not be in the streets. And it's not the first dangerous accident of an Uber self driving car where Uber was at fault.
Something does feel wrong about punishing them when the decision to put the car on the road in the first place was almost certainly not their own.
Though I agree Uber should be held accountable for it and it shouldn't be a token fine since the whole point of punishing an accident like this is to discourage them from occurring in the first place.
This sort of accident orchestrated by a group of people probably won't be gracefully handled by our legal system.
If one of those things caused the accident, the developer isn't to blame. Civil engineering has experience tracing liability from mistakes (and incentivizing prevention).
There's a reason why high pollution manufacturing moved to China. Because China was more willing to let their people die in exchange for jobs.
There's a reason sweat shops and slave labor happen more in some places than in others.