Uber Self-Driving Car That Struck Pedestrian Wasn’t Set to Stop in an Emergency

In no case would sudden braking be the cause of a rear-end collision. It's always the fault of the driver behind.

That's like creating HTML forms that only work with the common use case and crash spectacularly on unexpected input. Except that this time it's fatal. That's not the kind of software quality I want to see on roads.

This is beyond incompetence. There is a different level of software engineering when making a website vs making a pacemaker, rocket or flight avionics. You need the quality control of NASA, SpaceX or Boeing, not that of .. whoever they have running their self driving division.

Yeah that's not how you design safety critical software. This isn't some web service. Either you're wrong (let's hope) or Uber is completely negligent.

Source: I write safety critical code for a living.

Don't we have speech-to-text for this sort of thing?

https://www.nbcnews.com/tech/innovation/emails-show-arizona-...

"It might move out of my way" is no reason to get so close at such a high speed that you can't avoid it when it doesn't!

I'm not sure that'd be a huge issue. The vectors have to be intersecting first of all, which most vectors emanating from sidewalks wouldn't be, and then a little hysteresis would smooth out most of the rest.

This is also an issue with current production automatic braking system. One that was largely solved with on track testing, and on road testing logging false triggers with a driver driving. There's no need to risk lives unless you're just cutting corners to avoid the cost of a test track.

The way I see it,there is only way to make sense of a field where the most respectable R&D house (Google/Alphabet) limits their vehicle to a relative snail's pace while everyone else (including notoriously unethical shops like Uber) is taking a gung-ho, "the only limit is the speed limit" approach. That is to assume "everyone else" is cheating the game by choosing a development path that gives the appearance of being functional in the "rough but will get there with enough polish" sense while the truth is that it's merely a mountain of cheap and dirty hacks that will never achieve the goals demanded by investors.

The only reason a company would overlook such a simple safety protocol as "slow down until a potentially dangerous object is positively identified" is if their "AI" fails to positively identify objects so frequently that the car could never achieve what a human passenger would consider a consistent, normal driving pace. The same can be said for any "AI" that can't be trusted to initiate panic braking in response to a positively identified collision scenario with a positively identified object. The fact that they specifically wrote an "AI" absolving workaround for that scenario their software means the frequency of false positives must be so high as to make the frequency of "false alarm" panic braking incidents unacceptable for human passengers.

"As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path."

1. It seems like the classifier flipped state between pedestrian/unknown object/vehicle/bicycle; this seems like one of the well-known issues with machine learning. (I'm assuming the classifier is using ML simply because I have never heard of any other (semi-?) successful work on that problem.)

I suggest that the problem is that the rest of the driving system went from 100% certainty of A to 100% certainty of B, etc., with a resulting complete recalculation of what do to about the current classification. I make this hypothesis on the basis of the 4+ seconds when the car did nothing, while a response to any of the individual possibilities would possibly have averted the accident.

2. If the classifier was flipping state, I assume the system interrupted the Decide-Act phases of an OODA loop, resulting in the car continuing its given path rather than executing any actions. This seems like a reasonable thing to do, if the system contains no moment-to-moment state. Which would be strange; it seems like the planning system should have some case for having obstacles A, B, C, and D rapidly and successively appearing in the same area of its path.

3. Assuming the classifier wasn't flipping state, but presenting multiple options with probabilities, I can see no reason why the car wouldn't have taken some action in the 4+ seconds. (I note that the trajectory of the vehicle seems to move towards the right of its lane, which is a rather inadequate response and likely the wrong thing to do for several of the classification options.)

"According to Uber, emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior."

That's just idiotic and would be nigh-criminally unprofessional in most engineering situations.

What makes you think they aren't sent to VW all the time?

The place for the product folks to override safety features is the test track. If the feature didn’t work, they should have pulled the drivers because they were not trained to properly operate the machine.

If you give the “driver” training on a car with an autonomous braking system, then give them a car without it, that’s not on the driver. Someone was negligent with safety in regards to the entire program.

I’m not saying anyone needs to go to jail over this, but there do need to be charges IMO. Personal liability needs to be involved in this or executives will continue to pressure employees to do dangerous things.

The police are in no position at assert that, nor do they know whether or not Uber is guilty of negligence. Police do not bring charges and they're not running the investigation.

And partially-at-fault, on one hand, means there's fault on the driver side too, and on the other hand, is a judge's decision to make, not the police, no?

That should be irrelevant. Even if the pedestrian is jay-walking, it's still not legal to hit them. Further, having solid evidence that the car detected the pedestrian and did nothing to avoid her mitigates the pedestrian's responsibility, no?

Also, the "center median containing trees, shrubs, and brick landscaping in the shape of an X" sure looks like it should have some crosswalks, from the aerial photograph. What's it look like from the ground?

And that is why I am so mad at Uber. They are compromising the public trust in autonomous cars with their reckless release policy. And thereby potentially endangering even more lives, as we have to convince the public of the advantages of this technology.

This doesn't mean that Uber therefore did the right thing in disabling the system; it probably means that the system shouldn't have been given control of the car in the first place. But my point is that there is no readiness level where driverless cars will ever be safe -- not in the same way that trains and planes are safe. The driving domain itself is intrinsically dangerous, and changing the vehicle control system doesn't change the nature of that domain. So if we actually care about safety, then we need to be changing the way that streets are designed and the rules by which they are used.

I've seen a few people comment on the footage that they too would have run the pedestrian over, to which my only response is: I sure hope you don't have a driver's license [anymore]!

That depends, there could simply be no traffic behind you, which an experienced driver and hopefully and automated one would be monitoring.

Besides, there are many situations on the highway where an E-stop is far safer than any of the alternatives even if there is traffic behind you. Driving as though nothing has changed in the presence of an E-stop worthy situation is definitely not the right decision.

It hasn't shown that at all. It has documented beyond reasonable doubt that Uber should not be allowed to participate in real world tests of autonomous vehicles.

There are plenty of situations where people would fully accept a self driving vehicle killing someone but this isn't one of those.

This is an analogy that cannot completely map to cycling.

A fall at any speed from a bike is literally a potentially crippling or deadly scenario.

Uber had a fatality after 3 million miles of driving.

The mean fatality rate is approximately 1 per 100 million miles of driving.

It's a sample size of one, so the error bars are big, but it drives me insane that people are acting like the Uber cars are the ideal driverless cars of the imagined future, and are super safe. The available data (which is limited, but not that limited) is that Uber driverless cars are much, much, much more dangerous than mean human drivers.

That actually sounds like a really interesting concept, one of those ideas that seems obvious only after someone suggests it. What company is this?

Right now, in the Seattle area, we are basically seeing a new littering epidemic in the form of sharable bicycles being left to rust away, unused, at random places. If the bike could cruise to its next user autonomously, that would be really be a game-changer. "Bikes on demand" would turn bikesharing from (IMHO) a stupid idea into something that just might work.

Plus, the engineering challenges involved in automating a riderless bicycle sound fun.

What the Uber crash has shown us is mostly the willingness of people on HN to excuse Silicon Valley darlings even when they actually demonstrably kill people.

And "almost" is always a good idea when talking about a future that looks certain. Takes into account the unknown unknowns. And the known unknowns (cough hacking cough).

1.) Road safety -- as far as the current operating concept of cars is concerned (eg., high speeds in mixed environments) -- is not a problem that can be "solved". At best it can only ever be approximated. The quality of approximation will correspond to the number of fatalities. Algorithm improvements will yield diminishing returns: the operating domain is fundamentally unsafe, and will always result in numerous fatalities even when driven "perfectly".

2.) With regards to factors that contribute to driving safety, there are some things that computers are indisputably better at than humans (raw reaction time). There are other things that humans are still better at than computers (synthesising sensory data into a cohesive model of the world, and then reasoning about that world). Computers are continually improving their performance, however. While we don't have all the theories worked out for how machines will eventually surpass human performance in these domains, we don't have a strong reason to believe that machines won't surpass human performance in these domains. The only question is when. (I don't have an answer to this question).

3.) So the question is not "when will autonomous driving be safe" (it won't be), but rather: "what is the minimum level of safety we will accept from autonomous driving?" I'm quite certain that the bar will be set much higher for autonomous driving than for human driving. This is because risk perception -- especially as magnified by a media that thrives on sensationalism -- is based on how "extraordinary" an event seems, much more than how dangerous it actually is. Look at the disparities in sociopolitical responses to, say, plane crashes and Zika virus, versus car crashes and influenza. Autonomous vehicles will be treated more as the former than the latter, and therefore the scrutiny they receive will be vastly higher.

4.) So basically, driverless cars will only find a routine place on the road if and when they have sufficiently fewer fatalities than human driving. My assertion was a bit tautological in this respect, but basically, if they're anywhere near as dangerous as human drivers, then they won't be a thing at all.

5.) Personally, I think that the algorithms won't be able to pass this public-acceptability threshold on their own, because even the best-imaginable algorithm, if adopted on a global basis, would still kill hundreds of thousands of people every year. That's still probably too many. I expect that full automation eventually will become the norm, but only as enabled by new types of infrastructure / urban design which enable it to be safer than automation alone.

If you want to compare it with a car operating on cruise control you'd have to sedate the driver.

Or at least it should be. That's why there is a v_max that a car is not allowed to exceed and a faster or heavier car will have better breaks. And 70 km/h as here should be far away from v_max.

Redundant safety systems are a great idea, evidently Uber needed more of them, and integrating with the Volvo system might have been a reasonable option. It's silly to suggest that the integration would necessarily have been trivial, though. That's what I'm objecting to.

If any of the systems are vulnerable to any such false positives, and equipped to enable emergency braking to avoid them, even on the highway, it's not hard to imagine why they might be disabled, especially during testing phases.

I think it's fair to say that at this early stage in product development, there's probably no 'always right' answer for how to handle a given obstacle without considering locality, driving speed, road conditions, likelihood of false positives and negatives, etc.

Instructions from TCAS are near absolute in their priority. If ATC says to do something different, you ignore ATC and do what TCAS says. If the pilot in command says to do something different, you ignore the pilot in command and do what TCAS says. If somehow God Himself is on your plane and tells you to do something different, you ignore Him and do what TCAS says. Compliance with TCAS is non-negotiable, and the Überlingen disaster[1] is the bloody example of why it's that way.

Self-driving/autonomous-car systems need to have a similar absolute authority built in. If Uber disabled theirs because of false positives, it's a sign Uber shouldn't be running those cars on public roads.

[1] https://en.wikipedia.org/wiki/2002_%C3%9Cberlingen_mid-air_c...

The point is that so far most (all?) of these computer systems don't have a failure mode where it kills pedestrians, because their scope is very limited.

1. Regulators want aggressive breaking but carmakers want smoother driving

2. Manually tuning all the edge cases for cases where the software is uncertain what's happening will lead to fragile monolith black boxes

The regulators should only test for false negatives, where the car should have stopped, but did not detect the obstacle (false negative), because there, it is a clear threat to safety, and the car company's incentive, while definitely still present, is less pure, as the amount of false negatives is a direct trade off with the quantity of false positives (because it is a treshold: a minimum confidence level from which you decide that there is indeed something in front of the car and you need to break), which make driving more awkward for 99% of drivers

The charitable interpretation of this is that the industry believes that self-driving AI is a safety feature of greater quality than, say, lane assist or auto-breaking.

The less charitable one is that they find it too much work to integrate their AI to other safety systems. Which, to be fair, is really going to be a lot of extra work, on top of developping self-driving.

"emergency braking maneuvers" refers to an additional automated (software) system for automatically applying the brakes in an emergency (that's detected by that additional system).

I think the better interpretation is that the Uber system disabled another separate (non-Uber) system.

I really hate to defend this position because I feel terrible for her and it's quite obvious uber made some very bad calls here. (in fact I even used the very loaded word murder in another comment, see my post history). I'm definitely not arguing to put all or even most of the blame on the woman.

The intersection is indeed absolutely terrible. It makes no sense. You're not allowed to cross there, there are even signs which state it on both sides, referring to a crosswalk a minute walking up ahead. i.e., there is just no way you're allowed to cross there, despite the island in the middle having the cosmetic design of a walkway, you're not supposed to get on the island or get off it or cross the roads at that location. It's a terrible design choice, both inviting people to cross (in a dangerous spot), while also saying it's illegal. With better infrastructure planning you could have railing there preventing crossing, and no island at all.

That having been said, I can't for the life of me imagine crossing there and not seeing an oncoming car with its headlights on, assuming I was paying attention to the road I was illegally crossing. The fact meth is involved is a helpful possible explanation for why this kind of attention was not given.

You can check out the road on google street view, it definitely helps. I think you'd agree on two things, one is that it's not allowed to cross there and two that if you were to cross there, it'd be pretty easy to see cars. The reverse isn't necessarily true if you wear dark clothing at night.

Since it’s a no pedestrian zone, it has a higher speed limit posted

A higher speed limit causes a linear reduction in detection time, a sensor that detects a pedestrian 2 second away at 20mph will detects a pedestrian 1 second away at 40mph

Braking distance and impact energy are both quadratic and that’s part of how/why speed limits are defined

There are a lot of engineering issues that will arise from the need to cope with people violating limits and restrictions around autonomous vehicles.

I’m not defending uber here btw, they definitely jumped the gun deploying the system as described, just thinking out loud. There are decades of crashes that went into the driving code and road regulations, which humans kinda understand by relating to road design and getting cues from the environment that ais will need to learn

I didn't say you can't drive a car without being cautious.

I said you can't drive it without constantly braking the moment you detect an object, irrespective of what the object is doing. (e.g. moving into or away from the driving path).

i.e., just because an object was detected 6 seconds before impact did not mean the car ought to have started braking at that moment. It could be that the object was 200 feet away and moving away from the car's driving path, 6 seconds before impact. It'd be absolutely ridiculous to brake in that situation.

We have no information about this context, e.g. the car's data or determinations within the 6 second window. We only know it detected an object 6 seconds before impact.

It appears like the person I was replying to implied 'the braking distance was 180 feet, but the person was 380 feet away, thus uber could have prevented killing this woman had it not shut off the brakes'. In reality, the 6 second figure isn't relevant. What is relevant is the context that allowed a reasonable driver/AI to determine at a particular point in time, that the car should have slowed/braked. And we don't have that information yet. That's what I'm interested in.

The point I was trying to make is that I believe the noteworthy event is that a self-driving car struck a person, without even attempting to avoid it. That she was killed is a tragic twist to the story. I believe that even if she wasn't killed it would still be a noteworthy event because, again, the car did not attempt to avoid it.

I do not at all intend to diminish the fact that Uber killed someone.