Video suggests huge problems with Uber’s driverless car program
arstechnica.com
arstechnica.com
Wow. The other driverless car players should be all over this lobbying to shut Uber down. If Uber massively deploys a commercial service with subpar quality in order to "win", and then those cars start getting into accidents, the entire field is going to be delayed by 10 years. The general public is not going to just think "Uber is bad", they are going to think "self-driving cars are bad". Politicians will jump all over it and we'll see very restrictive laws that no one will have the guts to replace for a long time.
And honestly if that happens, that's probably what we would need anyway. If the industry doesn't want to be handcuffed they need to figure out some really good standardized regulations on sharing data with law enforcement, how to determine fault for self-driving vehicles, and what penalties there should be. That are fair and strict.
The alternative, they are not first to market, someone else is and immediately replaces Uber with a network of cheaper self-driving cars: a) Uber goes out of business b) They can somehow convince someone to license the tech or sell them the cars at a reasonable price, making them vulnerable and less profitable, with no market advantage.
Any engineer with this attitude needs to learn the lesson of the Therac-25. The issues in the Ars article are very similar to section 4 "Causal Factors" of the report[1].
> To get to that better software faster we should deploy the first 1000 cars asap.
Is that admitting that they do not have the "better software" and intend to deploy 1000 cars using "lesser software"? That's treading dangerously close to potential manslaughter charges if prove this willful contempt for safety to a court.
• Self-driving car kills child
• Hackers send self-driving car on wild ride
• Empty self-driving semi found in warehouse lot, GPS pirates make off with entire cargo
Creating the perfect self-driving car, with redundant systems, safety everything & so on will certainly help its safety records.
But it will also drive up the cost.
And put it out of reach for a lot of people.
If the goal is to save lives, the bar self-driving cars should be held to is what humans do driving today, not perfection.
> But zooming out from the specifics of Herzberg's crash, the more fundamental point is this: conventional car crashes killed 37,461 in the United States in 2016, which works out to 1.18 deaths per 100 million miles driven. Uber announced that it had driven 2 million miles by December 2017 and is probably up to around 3 million miles today. If you do the math, that means that Uber's cars have killed people at roughly 25 times the rate of a typical human-driven car in the United States.
I don't think there's enough evidence to say that self-driving cars are as safe as humans.
But that is human experimentation, something we as a culture generally agree is abhorrent.
This seems unreasonably optimistic.
First, this particular crash is an egregious counter-example. The car doesn't even seem to slow down when it first sees the pedestrian's foot. Nor does it try to swerve. This is basic stuff for a human driver, never mind more complex avoidance and risk mitigation a human driver can perform.
Second, we've had years of training various AI content curation algorithms on social networks, videos, blogs, etc - and the most advanced AI and search company in the world still can't keep adult-oriented conspiracy videos off of Youtube Kids. And while you might counter that content is a human problem, driving is too! Dangerous driving situations happen at the periphery of traffic rules, where someone is doing something the drivers around him don't expect. I've seen people run solid red lights, drive the wrong way down a one-way street, pedestrians start crossing when their light turns red, etc. Short of creating special roads for autonomous cars only - how does an autonomous car deal with all of this successfully?
I'm not sure which side of this I fall on just yet, but something strikes me here: you're assuming a human driver with no impairment (e.g. eyesight or fatigue) paying complete attention to what they're doing. We know that this isn's always the case (and could even be a minority!), so this doesn't seem to be a great argument.
It is irrelevant what average humans do. That is the current rule set for all drivers.
In fairness here: Google could do this so easily by throwing people at the problem. A whitelisted set of content producers could do this, with people dedicated to whitelist curation. Google can keep them off, and quite readily. Keeping that off automatically and cheaply is the issue...
On that front: automated content recognition that can parse the nuances between regular claymation Elsa and spooky claymation Elsa who talks about jamming things in her "happy spot" is AI-hard. These are not random videos, they are content tailor made and adapted to pass whatever filters are in place.
For data scientists there is a massive gap between working with concrete sensor data that can be managed reasonably, and undefined philosophical/moral/sexual boundaries in dirty noisy counter-programmed video content... One replaces our eyes and ears and reflexes, the other seeks to replace our fabulous brains.
To play Kalanick's adversary, he might be arguing for more real-world data collection. Tesla famously equipped most of their cars with more sensors than were required at the time of delivery, using the data to drive development of the Autopilot function that was later added to the cars.
Btw. all this of collecting sensor data and improving on this, is an original idea of MobilEye that they presented at conferences before the so-called Autopilot was available from Tesla.
He is clearly right about that. Human-driven cars are safety critical and already do fine without redundant brakes and steering. How many crashes are due to brake or steering failure? I'm guessing it's well under 10%.
Most human crashes are due to bad driving, and for computers it will be the same. I mean, even this fatal crash probably could have been prevented with better software. It's not like the brakes failed. They just weren't applied.
> To get to that better software faster we should deploy the first 1000 cars asap.
This is where he is totally mad.
Wow there goes that "safer than human drivers" argument.
It's a pretty unfair comparison, with 1 death on one side and over 30k on the other...
I would add that people represent (as a whole) various kind of drivers, including the non-expert/fresh licensed, the elderly, the too sure of themselves, those speeding or not respecting road signals (or driving in connection with a crime), the emotionally fragile, those under the effect of alcohol or drugs, those tired by not having had enough sleep, etc., etc.
The "model" for an autonomous vehicle should be instead a particular "perfect" subset, ideally it should replicate the behaviour of an hypothetical extremely prudent, healthy 30-something with some experience with safe driving, very familiar with the way his/her car handles, having had a nice night sleep, no use of alcohol or drugs, without any external pressure (to arrive in time, to do other chores apart driving) without a cellphone or tablet distracting him/her.
If we could find this latter kind of drivers (should they actually exist) and isolate from the statistics the amount of incidents they caused, that would be the reference benchmark.
Cars are killing by the millions now, so do the math, and I'm willing to tolerate a lot of deaths in the process, contingent on the deaths actually helping researchers and engineers reduce future faults. I would happily take a million deaths right now if it meant driverless tech instantly became available to everyone, but I think there's an upper limit to how many accidents can be realistically examined at once.
What's the alternative position? That you're for more road deaths in total? Are you also the kind of person who wouldn't pull the lever in the trolley problem?
Developing and testing the technology in actual test scenarios (build entire testing cities and fill them up with paid stuntmen and -women, if necessary) instead of in public, until you're able to prove that the tech is statistically at least as safe as human drivers. After that, you can continue testing in public, provided that you deal with any changes to your system responsibly and ensure that the public is not exposed to additional danger because of your testing.
This is actually very common in the software development world. For production-critical systems, companies go to great length of creating a staging environment as realistically as possible, but fully separated from the production systems. You may get around that effort if the only damage you can potentially do is people not being able to see stupid cat pictures on a social network. But sorry, for life-critical tech, the move-fast-and-break-things approach is irresponsible bullshit.
> I would happily take a million deaths right now if it meant driverless tech instantly became available to everyone
What if your own death is guaranteed to be among them? Still "happily taking" it?
For the sake of the discussion, let us assume that this is hyperbole and that "happily" was an infelicitous word choice. That said it raises a few questions:
- Uber and the other companies presumably are developing this technology for their own private benefit and do not plan to make it "available to everyone". In which case, how many deaths are acceptable?
- Should they be successful and succeed in developing a safe autonomous driving system should they be compelled to make it "available to everyone"?
- Are these million doomed citizens volunteers with informed consent or are they to be struck down unawares at random?
- What if the technology proves more difficult to develop than you anticipate? Suppose that after one million are sacrificed for the greater good it is improved but not yet good enough. Should we then continue with another million or should we abandon the project after only one million fruitless deaths?
- Assuming success, should we then ban human driven vehicles completely?
[edited for formating]
If your faith is strong enough, you'll surely be welcome if you sign away all of your rights and serve as irrational traffic participant in a city where self-driving cars are tested. You could run through the city all day and produce difficult safety-critical situations by suddenly walking in front of them.
My faith isn't strong enough for doing that, so I expect lawmakers to protect me from companies like Uber who apparently think it's okay to basically make me their guinea pig in their public experiments. But I'll applaud you if you decide to take the risk on yourself for the (potential, unproven) advancement of the human race!
I am pretty convinced that computers have the potential to outperform humans in this, but I'm not really interested in hashing out the debate here; the arguments have all been made more elegantly elsewhere. Though I don't know why you keep ascribing 'faith' to me. I never brought that dirty word up. I reasoned my way into my positions.
In that case, what do you base your assumption on that self-driving cars will be able to deal with the complexities of todays' traffic in a significantly safer way than humans? Because even though you might not bring that word up, if you just assume that to be the case because you expect technology will one day be able to make that wish come true, that IS "faith" in technology. Whether you like that word or not doesn't matter.
The rationale response for individual cities is to say: "Do your risky testing elsewhere, thank you very much. You may come back here once you are as safe as our average driver."
> I believe with perfect faith in the coming of the Messiah; and even though he may tarry, nonetheless, I wait every day for his coming.
Consider if there was a new lottery and you weren't sure what the odds of winning were. You play it three weeks in a row and the third time you win a million dollars. Conveniently, no one else tries the new lottery yet.
Does it follow then that the odds of winning a million dollars are 1 in 3? Or should you play it a few more times before you declare to all that one in three plays will make one a millionaire?
Would it have been fair if Uber last week were to declare that they have a 0% probability of pedestrian deaths, since they'd never had one yet?
The goal of these statistics is to predict future outcomes. But with such a small data sample, you cannot fairly predict the future- just as in my lottery example.
(I'm guestimating on some of those numbers, but the should be somewhere in the ballpark.)
I'm not defending Uber here- I'm defending statistics!
With more data, we may discover that Uber cars are 100X worse, not just 25X. Or we may discover they're better. But we don't have the statistical power to make that estimate when we've only seen the event happen once.
In this we agree.
It only took two incidents to shut down the Concorde program.
Assuming that accidents are independent, we can model this as a Poisson point process. If the accident rate is 1 per 100M miles and Uber has driven 3M miles, the probability of there being zero accidents in that time is P{n=0} = ((λt)^n / n!) * e^(-λt), where λ=1/100M and t=3M. Doing the math, it seems that's 97.04%.
So, yes. It is possible that Uber's accident rate is 1 in 100 million. If so, this incident would fall in that remaining 3%. It's unlikely, but possible.
One thing we can say about the woman killed the other day by an autonomous Uber, is that unlike the other ~40,000 killed on America's roads over the past year, her's was not in vain.
Every day that we delay the widespread deployment of this technology, it's another 100 or so people dead. Of course, the public is unlikely to see it that way. They see one death as a tragedy, but 40,000 is just a statistic, business as usual, nothing to get excited about.
Trading lives to save on R&D time would violate professional codes of ethics in literally any other industry.
This is the real trolly problem when it comes to the ethics of developing self driving cars.
If Uber needs data they could have driven manually. Obvously their obstacle tracking is bad.
Uber is doing this for money, so is all the other companies even if there are some potential huge collateral benefit for the human race from that. It's definitely not the goals of the companies regardless of any PR talk. So when you gamble with people's life for money and fame you should go to jail for a long time executives or engineers alike.
The statistics are just being used to sustain corporate greed in my mind and we should not let them. Self driving cars has lots of potential to save life, so are other techs. Does't mean that all sense of responsibility and ethics goes away just because of the potential.
The current procedures for clinical trials are the result of decades of experience, where mistakes (and yes, occasionally shortcuts driven by greed) did result in avoidable deaths of trial participants.
For an example of how the pharma industry deals with this 'safety first' versus 'stifling innovation' dilemma, read this article: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1526936/
The inevitable result of this crash is driverless vehicle testing will get regulated more like drug trials...
On the other hand, there are widely used, cheap, and efficient alternative to self-driving cars on the roads today: human drivers, public transport, carpools, etc.
If you want to make analogies, I think the self-driving car accident is more like if an elevator company accidentally crashed an experimental high-speed elevator in a shopping mall. I think it would be grossly negligent on the part of the company to test such an unproven device on the general public, especially if people’s lives are being put in a position of risk which they did not to worry about before.
Such a thing would be entirely unthinkable in the pharmaceutical industry of today. So if this comparison suggests anything, it is to much more strictly regulate self-driving car development and testing!
Uber should be banned from doing AI research on public roads
Bayesian approach: To make the math really simple, let's assume a discrete prior on Uber's death rate. Say 33% that Uber's cars are much safer than humans (0.1 deaths per 100M miles), 33% that they are equally safe (1 death per 100M miles), and 33% that they are much more dangerous (10 deaths per 100M miles). After observing one death at 3 million miles, your posterior is should update to {safer: 1%, equal: 11%, more dangerous: 88%). This is a substantial shift in confidence.
Math: http://www.wolframalpha.com/input/?i=(1-10%2F100)%5E2*(10%2F...)
Frequentist approach: Let the null hypothesis be that Uber's self-driving cars have the same death rate as humans - 1 death per 100 million miles. The odds of Uber killing someone within 3 million miles is about 3%. Therefore, we can reject the null hypothesis with a p value of 0.03. One positive "data point" is statistically significant.
Statistically, one death after 3 million miles is not proof that Uber's death rate is higher than 1 in 100 million miles. But it is statistically significant, in both a frequentist and Bayesian framework. You have to get really, really unlucky to have a death at 3 million miles if your death rate is 1 per 100 million miles.
Bottom line: This collision isn't proof, but it's strong evidence. (To go along with all the evidence from crash rates, disengagement rates, engineers working at these companies, and the video of the crash itself.)
They let a car on the road that couldn’t even stop at a red light ffs
https://www.theverge.com/2017/2/25/14737374/uber-self-drivin...
The fact that they were allowed to deploy in Arizona after this is really regrettable
And it’s totally unsurprising that Uber “got the first kill”
A proper comparison in this case is comparing passenger car death rates.
And then you need to factor in other conditions, such as the fact that weather was clean, and that you should be looking to compare pedestrian/bicyclist deaths, and you see that this incident already throws out wack the death rate for autonomous vehicles.
Taking it a step farther, I’d expect that road conditions in the Uber tests so far have been more benign than average city driving. Less bad weather, straighter roads, etc.
The story in the linked article about an Uber whipping through an intersection at 38mph next to two stationary lanes, seems sufficiently conclusive to me that their self-driving system is not ruled by a sense of caution.
Here in the UK, we have speed limits, but the rules of the road also call drivers to consider "appropriate speed" - you slow down in situations where you might have to react with very little warning. This ought to be extremely easy for an automated system - it can measure its braking distance with high accuracy, it can measure distances to objects around it with high accuracy, it can determine exactly which areas of the world around it it can't see, that could pose a threat with relatively low warning, so just fucking slow down.
I've long been bearish on full autonomous driving because I consider there to be so many corner cases in real world driving where ad-hoc non-verbal communication is required to solve traffic flow, that the computers would never catch up. Now I wonder if their solution is to just plough through every problem at 98% of the speed limit and then disclaim responsibility.
Set "too tightly", it will also have you slowing for every car approaching a stop sign from a side street.
Cars that randomly slow out of an excess of caution are also a hazard to other road users. Don't believe that? Go drive for a month and set a series of alarms on your phone every 5-10 minutes. Everytime the phone goes off, abruptly slow to half of your prior speed. Do you think you'd make 1,000 miles without causing a road hazard or collision?
I believe that might mean that autonomous vehicles are not yet ready for road testing if that is commonly required by the current state of the art. (I last worked on autonomous vehicles in 1991; ours was entirely rules-based and we tested on public roads in addition to private tracks. Ours was bad enough that the human driver hovered over the red E-shut button and was always paying attention. It was harder work than just driving the damn thing yourself, but we had to test in order to make progress. I'm sure loads has improved since then.)
I also don't think that zero fatalities is a realistic goal nor is it the standard that should unduly inhibit progress. People have been dying in transit on foot, on horseback, on bikes, and in cars. This is a version of the trolley problem. I don't mind and in fact actively prefer a system improvement that allows 100 deaths while saving 500, even if the 100 is entirely disjoint from the counter factual 500.
In this specific case, I believe the autonomous car allowed 1 death that would have also been allowed by a human driver in the same circumstances, so it's a push.
This is because stopped vehicles often have people come out of them to fix the vehicle, and too many cops, tow operators, etc, were getting killed by careless motorists.
Guess this is news to you (and uber), update your driving style please for the sake of first responders!
For example, unnecessarily stopping in a middle of a highway is extremely dangerous, especially if visibility is limited or roads are slippery.
Looks like Uber has attracted Levandowski due to his cultural fit.
"To get to that better software faster we should deploy the first 1000 cars asap. I don't understand why we are not doing that. Part of our team seems to be afraid to ship."
And from another email:
"the team is not moving fast enough due to a combination of risk aversion and lack of urgency"
On the other hand, redundant steering and braking seem like probable overengineering. Brakes are already somewhat redudnant (dual section master cylinders were common in the 70s and are almost certainly in any modern vehicle), and better software could periodically verify they're working and if not, coast to a stop. Steering failure could be handled by engaging the brakes. Simultaneous failure is likely rare and catastrophic anyway -- losing a wheel and having the brakes pressure go with it can happen, and when it does, you put on your blinkers and hope you come to rest in a safe manner.
You absolutely need a system to ensure a controlled stop in any type of critical failure in ability to control the system. Assuming you have that, it seems reasonable to regularly verify the controls are functional (jiggle the steering, modulate the throttle, gently tap the brakes) every so often, and rely on your controlled stop procedure in the event of failure.
I do have the common sense to avoid safety critical systems, thanks; however armchair engineering is a national sport.
And then, if their code results in a death, they are liable and can have their license completely revoked, and they would be unable to work on self driving cars again.
https://insight.ieeeusa.org/articles/professional-licensure-...
That's the whole process. Repeat that over a long enough period of time and you tend to select for a more competent, safety-conscious group of engineers.
Over the long term, its survival of the fittest. Reckless engineers have safety incidents and get barred from both working and from supervising new engineers. So, it tends to be the safer engineers who get to pass on their work culture to the next generation of engineers.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
- If bad code makes it into production, that is a systemic failure not an individual one (Why didn't the bug get caught in code review, QA, etc.)
- No one is going to want to work on a project where a single failure can taint their career.
- What if I use a 3rd party lib and that is where the bug is. Who is at fault then? What if the code isn't buggy, but I'm using it in an unexpected way because of a miscommunication? If I am only allowed use code that I (or someone certified has written) development is going to move at a snails pace.
- What if I consult with an engineer who doesn't have a certification on a design decision and the failure is there, who is at fault?
- What if the best engineer on the project makes a mistake and ends up banned? Does he/she leave the project and take all their tribal knowledge with them, or are they still allowed to consult? If they can consult, what stops them from developing by proxy by telling other engineers what to write?
Not to be a dick, but this is an awful idea that would basically kill the self driving car.
Yes, there are reviews, QA, and all of that. So, yes, there is no single person responsible (exceptions apply).
But there is no excuse for using 3rd party libs. Just don't use it. If you not know: do not use it.
That is the reason, why certifications are for. The same rules apply for medical and other areas.
Wait, what? That goes against one of the core benefits of open source software--that having many eyes on a problem decreases the risk of bugs. I'm willing to bet that if Uber had to implement their own machine learning/vision libraries from the ground up, there would be significantly more issues.
Certification etc is about process. Open source code can be used in a safety critical product, but it must audited and confirmed against the system requirements.
The problem with that is, nobody audits code, if it is working just for fun. And even if it is buggy, then most people look for bugs in their own software and then they work around, so that the original piece is not modified.
We have this seen in many open source projects. Remember all the obvious, mostly security related, bugs that weren't uncovered for years. They weren't uncovered because everybody thought: "huh, that is hard. I assume that other more experienced than I will have reviewed it, so I will trust it."
The thing with certification is, that it is required that it is really reviewed. That there is a guarantee that it is reviewed. That there were people with a different mind set, with different background have reviewed it and as such have brought in their own view.
Certification does not guarantee that something bug free. It guarantees only, it is reviewed. Open source has no guarantee that it is reviewed. There is only hope, that someone has reviewed it.
Pretty much everything in development relies on the work of other people. I used 3rd party lib just as an example, but what if it's in the framework or even the language that an engineer uses, who would be at fault then? You can't expect every developer to have gone through the entire source code for whatever language they are writing in.
Sciences build on each other and and after a certain period of time you have to take things for granted in order to keep moving forward.
> The same rules apply for medical and other areas.
No, they don't. Doctors kill patients all the time and they aren't banned from medicine for it. There is an investigation, they make sure it wasn't intentional and there wasn't any gross negligence and that this isn't a repeating pattern, if none of those are the case they see what they can learn from it and move forward in hopes that what they learn can help other doctors.
I agree dith parent: never use copy paste from internet in safety critical SW, anyway it most probably isn't designed for your use cade. Personally I always have been disapointed by copy pasting stuff, it was always buggy somehow. In the end I always reimplemented it from scratch by reading the theory
From what we seen so far the Uber car failed to detect an obstacle, we also had Tesla crash where the car did not seen a truck so it is obvious that there are some major issues that are not tested for. They need to have better tests and maybe get better security drivers inside the cars so they don't text on the phone on the job.
Does that demonstrate that the cars are safe? Even a little bit?
To me it just demonstrates that a grunt is on the chopping block for what amounts to systemic failure.
In safety-critical fields, setting a much higher quality bar than the regular 'it seems to works, the tests pass' seems perfectly rational to me. We can now write provably-correct C compilers (CompCert) and OS kernels (SeL4). There's no excuse for not putting similar levels of effort[0] into something as safety-critical as self-driving cars.
[0]: Note that I'm not advocating for "provably-correct self-driving car software" (that may not be the right approach, as a formal spec is likely unrealisable), but find the argument that "it's ok to write buggy spreadsheets, so it's ok to write buggy self-driving cars" to be morally unacceptable.
"The engineers employed by Jack D. Gillum and Associates who had "approved" the final drawings were found culpable of gross negligence, misconduct and unprofessional conduct in the practice of engineering by the Missouri Board of Architects, Professional Engineers, and Land Surveyors. Even though they were acquitted of all crimes that they were initially charged with, they all lost their respective engineering licenses in the states of Missouri, Kansas and Texas and their membership with ASCE.[22] Although the company of Jack D. Gillum and Associates was discharged of criminal negligence, it lost its license to be an engineering firm." - https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...
Just like it relocated its testing to get a more "business friendly regulatory environment".
For a mostly topical example of such rules: https://www.dmv.ca.gov/portal/dmv/detail/vr/checklists/outof...
Likewise if you knowingly observe anyone else in your company breaching safety/regulatory guidelines then as a professionally certified engineer you have a legal responsibility around ethical disclosure.
See: http://www.professionalengineers.org.au/rpeng/ethical-commit...
I do not know how things work in the US but in Australia these rights are protected by law. The company legally can not fire an engineer in this situation.
Professional engineers (note the lowercase 'e') are usually not protected in the U.S. -- they're regular employees whose profession happens to be engineering.
If you want error-free software you need a blameless culture based around process, not individual ownership of code. It should not even be possible for an error to be one individual's mistake, because by the time it hits the road it should have gone through endless code review and testing cycles.
That'd be extremely foolish. And regardless of the dumb things the previous Uber CEO has done in the past and the big deal people are making over a $150 license, they have still hired some of the best engineers in the world.
You basically have to find the brightest-of-the-brightest to build AI... and Uber pays very well and puts plenty of effort into recruiting that talent.
Not to mention the massive PR and monetary risks that are inherent in killing people with your products. That would make any company highly risk-adverse.
https://jalopnik.com/safety-third-is-the-running-joke-at-ube...
1) They make it appear that Uber is a car manufacturer.
2) Even though Uber has not been determined to be at fault, the author seems to want to make it that way anyway.
1) The physics of driving isn't random, so it could be said that there are no accidents in autonomous driving, only oversights.
2) It would set a minimum performance level by making it prohibitively expensive to have a dangerous car. Those who test responsibly would have a low enough injury rate that they could deal with the risk by taking out suitable insurance.
3) It would provide a strong incentive to make the best car possible and not to take expedient shortcuts.
4) Over time automatically liability would become irrelevant if it asymptotically forces the injury rate to zero.
5) We have an historic opportunity to create a culture that will eliminate the danger of cars. It might have an increased short-term once off cost, but a huge long term payoff in the reduction of health costs and human misery. If we miss this opportunity we will be stuck with the long term cost of an industry that will be competitively driven towards poorer performance, potentially against the will of the majority of players, by the actions of a few.
Additionally, if we train self-driving cars to always give way to pedestrians who even look like they might cross the street, they’re going to have a heck of a time getting through cities. Kids are going to learn that they can trigger a squealing emergency stop by lunging towards the curb - great fun!
What I think WILL happen is that autonomous cars will have to buy blanket insurance policies that cover their entire fleet. High accident/fatality rates will result in high insurance premiums, which will put bad actors out of business.
Where I live in California, there are two major streets with crosswalks in regular use, with speed limits of 40 mph or higher (and travel speeds of 50+ mph), within a quarter mile of my home.
Taking into account optimal (computer driven) car stopping distances, the maximum acceleration of a person and typical pedestrian densities, I don't think getting though cities would be an issue, especially outside the CBD. Even in areas with very high pedestrian traffic cars will be able to get though. In support of this argument, I offer today's "shared" pedestrian zones, where cars and people mix. Pedestrians have right of way, cars are limited to 10km/h, but the cars manage to get though without injuring anyone. Cars would naturally do high speeds on main roads with low pedestrian density and lower speeds (with very short stopping distance) at high pedestrian density.
Why should children have a propensity to intentionally jump in front cars above and beyond anyone else? That's bias.
If anyone intentionally jumped in front of a car then it would be covered by the exemption that I proposed: that the car would not be liable if intent could be proved. Based on the car's sensor logs it would be pretty easy to prove that someone intended to get hit. If the car managed to stop and the person ran away it would then be relatively easy to track that person down based on the logs and charge them with a crime. In any case, I think that that is a hypothetical situation which is unlikely to occur. For the vast majority of people self preservation would trump the desire to cause trouble by putting oneself in a terrifying and life-threatening situation, so I think it would be negligibly rare.
Changing the average travel speed from 45 kph (the city-wide speed limit in New York) to 10 kph would be a disaster.
And what alternative do you suggest? Not braking?
If I see a kid running towards the road, I'm going to slow or stop my car. I'm not going to think "oh it's just a kid being a jerk, they'll definitely stop before the road" because I don't want to run the risk of squashing a kid.
Therefore, I behave exactly the same as a cautious AI would be expected to behave.
So, why aren't kids running to the edge of the sidewalk when I'm driving?
Solvable by reducing speed sufficiently. It's reasonable to expect the car to avoid a man travelling at 10kph from any off-road blind-spot, and a car travelling at 100kph from any on-road blind-spot.
> Additionally, if we train self-driving cars to always give way to pedestrians who even look like they might cross the street, they’re going to have a heck of a time getting through cities. Kids are going to learn that they can trigger a squealing emergency stop by lunging towards the curb - great fun!
If you sprint to the curb in front of traffic today, drivers will stop/swerve. Almost certainly illegal too.
My interpretation of the parent post is that more responsibility can be put on the car to avoid accidents, than is currently the case today. Hence greatly increasing road safety. It sounds great!
Likewise, I can be standing still with my toes on the curb, and then lunge into the street. Should a self-driving car assume that every pedestrian standing at a crosswalk could walk into traffic at any moment, and slow down accordingly? Again, that’s not what human drivers do.
There are a number of surface streets near my house with speed limits of 45 mph, and crosswalks every 1/8 mile or so. Requiring cars (autonomous or not) to avoid any possible pedestrian incident at every such intersection would be a disaster for traffic throughout and a huge step backwards.
A human should be aware that these pedestrians might enter the roadway. The human should perceive those pedestrians as a risk, and be ready to take action.
> Should a self-driving car assume that every pedestrian standing at a crosswalk could walk into traffic at any moment,
Yes.
> and slow down accordingly?
This doesn't follow. The car don't need to slow down. It does need to be ready to perform an emergency brake.
I fully agree that autonomous vehicles can and should do everything they can to avoid accidents. We are in violent agreement there. However, I also think that if we set some kind of unrealistic standard for safety, then we are going to make self-driving vehicles completely unappealing to everyone, because they are going to drive like a cross between my grandmother and a startled squirrel.
This is not a new idea of mine - the issue of the too-polite autonomous car has been extensively studied and reported on. See https://www.google.com/amp/s/mobile.nytimes.com/2015/09/02/t... for just one example.
That genuinely might not be a bad thing, at least in the initial phases. They should be ruled by an abundance of caution until we're sure they can actually make more aggressive decisions.
Uber seems to be defaulting to maximum aggression from the outset, which is hardly surprising from them, but seems extremely over-confident (in fairness, like most new drivers are ;)
The point of defensive driving is that you can't rely on other people not to be incompetent.
You have a green signal at a traffic light. That does not mean "go", it means "proceed with caution".
Isn't that already the case (and arguably they'd get more satisfaction out of it in some cases by seeing an annoyed driver)? Further, I'd say that's what _should_ happen -- if someone looks like they are going to enter the road, you stop.
If you enforce your right of way and kill somebody, that's manslaughter in my book.
They are on the road with conditions because what they are doing is somewhat experimental still. There is a safety driver for a reason that did not respond. A human driver may have collided but would have responded and potentially avoided a fatality (if not a collision). The benefits of autonomous driving completely failed on all counts in this case, which imply that being on a public road is far to early for Uber - suggesting some fault to lie with Uber or the regulators.
I'd be fairly surprised if there's any real appetite at Uber to continue with this now. It was never anywhere near their core competency.
Maybe not directly, but is Uber's current business model sustainable without some form of self-driving technology replacing their human drivers?
3 years ago Uber hired ~50 specialists from CMU to work on autonomous vehicles. I'd call that a core competency.
https://www.theverge.com/transportation/2015/5/19/8622831/ub...
"Indeed, it's entirely possible to imagine a self-driving car system that always follows the letter of the law—and hence never does anything that would lead to legal finding of fault—but is nevertheless way more dangerous than the average human driver. Indeed, such a system might behave a lot like Uber's cars do today."
It doesn't matter if Uber makes cars that are technically not at fault, if they're mowing over pedestrians at a rate significantly higher than human drivers then they should never be allowed on public roads. People mess up occasionally. The solution is not an instant death sentence administered by algorithm.
And whether Uber makes the entire car or not isn't germane to the discussion. They are responsible for the safety of said cars, which is what we're discussing here.
Your second paragraph emphasizes my point. You,too, are stating that Uber is a car manufacturer, in whole or in part. Does Uber manufacture any parts of this car at all? The impression the opinion piece gives is that Uber manufacturers cars.
It seems unlikely that the Police will find any fault because they probably don't want to have to file a criminal charge against the driver, but that is who it would go against if there was fault.
There was a case some years ago in Boston where a subway rear-ended another one because the operator was texting. The driver was fired and probably would have been prosecuted if there had been fatalities. Taking eyes off the road for this long seems insane to me.
If you really cared about safety, there are far more immediate and impactful solutions then spending billions on self-driving cars. If they came out and said that they were doing it to make money or make driving easier, it would have carried more weight. But you just can't trust a word this company says.
* Flouted Taxi regulations
* Living in legal gray zones in regards to contractors vs employees
* Designed a system to avoid law enforcement
* Performed shady tactics with its competitors
* Illegally obtained the private medical records of a rape victim
* Created a workplace where sexual harassment was routine
* Illegally tested self-driving cars on public roads in California without obtaining the required state licenses.
* Possibly stole a LIDAR design from a competitor
Now their vehicle killed a pedestrian in a situation that the self driving vehicles should be much better than humans at (LIDAR can see in the dark, and the reaction time of a computer is much better than humans.)
Uber has exhausted their "benefit of the doubt" reserve. Maybe, they need to be made an example of with massive losses to investors and venture capitalists as an object lesson that ethics really do matter, and that bad ethics will eventually hurt your bank account.
edit: wow this triggered some people. somehow 'if they are at fault they should be punished' got interpreted as 'they are not at fault and should not be punished'
There have been cars on the market for years now that can detect a dangerous situation ahead, even in the dark, and auto-brake. If they haven't caught up to several-years-ago's consumer model of car, then what on earth are they doing putting these things on public roads?
The point is that it is to early to test this cars on public roads. The fact that self driven cars will kill less is just a hope, we may never achieve that(there is no proof we can do it with current tech) or it will take more time, and we need some actual numbers that are not tampered with before letting this self driven cars on the road, the security drivers seem to not pay attention so this car testing are a risk.
> But zooming out from the specifics of Herzberg's crash, the more fundamental point is this: conventional car crashes killed 37,461 in the United States in 2016, which works out to 1.18 deaths per 100 million miles driven. Uber announced that it had driven 2 million miles by December 2017 and is probably up to around 3 million miles today. If you do the math, that means that Uber's cars have killed people at roughly 25 times the rate of a typical human-driven car in the United States.
We have a sample size of 1, granted, but it's not looking very good. At the very least they were expected not to be less safe than humans.
Almost all the miles driven are going to be in near-ideal circumstances (daylight, no rain, good road surface, driver familiar with normal road traffic conditions and drives the route regularly). I have nearly no insight into the uber death, but I gather it happened at night. It could easily be that humans are also an order of magnitude more dangerous at night.
Suppose, as a massive oversimplification, Uber's self-driving cars crash with some constant probability P for every mile driven (i.e. a Bernoulli process).
We now have learned at least one thing with absolute confidence: P > 0.
The first mile driven before the accident, of course, also showed P < 1.
But beyond absolute certainty, we also have a better idea of the actual value of P. (Intuitively, the longer we go without a crash, the lower we suspect P to be, and for every crash, we increase our estimate of P).
If there was only one crash in 3 million miles driven, this is evidence for values of P near 1/(3 million), and evidence against values of P far from it.
Is it strong evidence? Nope! But it's evidence!
Your pedantry has managed to upset me and I would encourage you to be a little more understanding of people using language in the way it is used outside of the world of mathematics. Walking into a practical discussion of safety with an existence proof of all things is disrespectful of the fact that lives and enormous quantities of human attention are at stake.
Obviously, technically everything is evidence of something. I know that. Using the language in that sense is not going to help.
Something happened that just shouldn't be possible, with Lidar and half-competent AI - which we use in good part because cameras don't handle low-contrast nearly as well as human eyes, say at night.
> It could easily be that humans are also an order of magnitude more dangerous at night.
Given that the actual circumstances of the death (pedestrian crossing left to right in a large road with the car in the rightmost lane) was a case that autonomous cars should've been much more equipped to deal with, and the Uber clearly failed, it's not promising. Autonomous cars were supposed to be better than humans; this particular one does not seem to be, given that as a human I can pretty clearly see someone crossing from the far side of the road.
i'm not sure that's the right way to look at it (that uber is a sample of 1, or that the death is a sample of 1).
in this case, the metric is deaths per mile, so there are purportedly 3 million samples for uber self-driving cars, with one positive (negative?) result in that sample. you need so many samples because the positive observation rate is expected to be very low (as evidenced by the 1.18 deaths per 100 million miles driven by human drivers).
if you assume the death rate is roughly the same, you can (roughly) estimate the expected error or confidence interval with the 3 million sample size versus a 100 million known rate for human drivers. as more samples are gathered, the confidence interval gets tighter: if the confidence interval currently stands at 80% with 3 million samples (made up numbers), it might go up to 85% with 6 million samples.
It will be interesting when Volvo pulls back from that relationship, because even the uptodate emergency brake assistent should have catched up. Except Uber as deactivated those in favor for their own technology (because it is superior).
EDIT: German Automotive Club did tested full sized cars in 2016, which means decaded old technology in IT. The results where that the Subaru Outback was the only one detecting persons in the dark. But was very poor with biclysts. https://www.adac.de/infotestrat/tests/assistenzsysteme/fussg...
Self driving cars are currently in that state where they're always in accidents but never technically at fault. When individuals have this behavior patter their insurance company drops them because if they're so frequently present when shit hits the fan they're a time bomb from a risk perspective.
Edit: meant to reply to parent, oh well.
Maybe testing of autonomous vehicles should be done off public roads (at least at this stage of development).
The way it would work would be: the human is driving and the software is, at the same time, watching the driver and figuring out an action to take. Every time the driver's and the software's behavior differ, is logged and analyzed to figure out why there was a difference and who guessed better.
But the way testing is currently going on, it seems millions of miles are wasted where nothing happens and nothing is learned.
Yeah, that doesn't work though. Basically because you would need to have an excellent situation representation to really understand the drivers' reactions to outside events. But that does not exist.
Perception and situation representation are key to mastering the driving task, and they both differ greatly between humans and machines.
This is the #1 reason I'm sceptical about self-driving cars becoming ubiquitous any time soon. Clearly they have potential advantages over a human driver in terms of not being tired or distracted, better sensors and better reaction times, but their judgement in any given situation will always be a function of some predetermined inputs. It's a brute force approach.
Until a self-driving car can recognise a pub door opening around throwing-out time where a drunk patron is about to stumble out into the road from a hidden position, or that it's about to pass a park and a nearby school just finished for the day so kids will be kicking balls around and running across the road to join their friends, or that the recent weather conditions make black ice likely and the cyclist it's about to pass doesn't seem very steady, and take corresponding actions to reduce both the risk and the consequences of a collision, it's going to take a lot of brute force to outperform an experienced and reasonably careful human driver.
In short, reacting to an emergency 100ms faster than a human driver is good, but sufficient situational awareness and forward planning that you were never in the emergency situation in the first place is better.
In fairness, I will concede that pattern recognition is crucial to intelligence. I would clarify my earlier position by saying that pattern recognition alone can only get you so far. Intelligence is the process of taking data, recognizing patterns, constructing multi-layered concepts and models from those patterns, and then being able to simulate and extrapolate potential outcomes based on varying inputs.
The inability of modern AI to build complex concepts from they patterns they observe is why I know to slow down my car on a Halloween night in a residential neighborhood and a self-driving Uber doesn't.
- on Halloween there are more children on the road than usual - children are more likely to disobey traffic laws
It’s quite possible that waymo cars are already capable of this.
Some argume that it is okay, because it will decrease risks in traffic in the long run. This is not a valid argument to allow on road bug-testing, as there is a lot of medical research that we as a society don't allow because of ethical concerns, even some research where the risk of death is essentially zero. Applying research ethics to the Uber situation, Ubers vehicles would under no circumstances be allowed on the road until it could be proven that they were at least as safe as all vehicles already operating on the road.
So while the technique suggested might not work well to solve the problem of safe autonomous cars, the more dangerous alternatives should absolutely not be allowed.
"win-at-any-cost" and "second place is first looser" (sic) do not cohere with safety.