Uber disabled Volvo SUV's safety system before fatality
sfgate.com
sfgate.com
I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way.
> having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.
That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature.
Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems".
I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.
I'm so glad they're not testing their cars in my state. This makes me wonder if they even do basic safety engineering like FMEA or if they just wing it.
There's 2 chefs.
The collision avoidance system? The fail-safe that's meant to ensure the car doesn't hit something when the human driver screws up? The one that works with an entirely separate autonomous intelligence (aka, the human) controlling the car? The one explicitly designed to trigger emergency brakeing even if the primary controller is trying to do something else? The kind of system installed on many other AI controlled cars to avoid this exact type of disaster?
That backup system?
Look, it's entirely possible that for some reason this specific safety system conflicted with the current version of Uber's system, but we have zero data to suggest that, and if so, it's something specific to those two systems in that particular vehicle (and suggests a severe issue with Uber's system to boot). There's no general issue here.
> There's 2 chefs.
Yes, and sometimes one of them takes over from the other one. That's how this works.
There are plenty of intentional maneuvers required to integrate with real-world (ie. non perfect) traffic that trigger these systems.
They’re designed to take hints from human input (so for example, LKA responds less strongly to forceful unexpected inputs than a lack of expected input).
There is 0 chance the system wouldn’t interfere with a self driving.
Not to mention many of those safety systems include lockouts on the CAN bus which make removing their safety module a first step to even add your own inputs. I know that’s how it works on my Volt with a Mobileye sensor
If your SD system is supposed to be solving 100% of the same problem anyway, it's best to gut the software you don't control and work with the hardware as directly as you can. The notion of building atop a lower-level abstraction makes for great academia but doesn't work with industrial realities.
So that would, if the automated tech from Uber would have detected in time, it would have reacted like a human driver and the pre-installed system would may have not noticed it. But in case, it would have reacted as a backup.
There is no warning.
Not trying to take sides here, but the last part strikes me as "citation needed".
Do you know of anyone who's died that way?
This further reinforces my point, I think. :)
While yes - it would also force Uber to permanently partner with Volvo on technology they can't control. 'Uhuh, my Ubers a Chevy not a Ford - it could kill me because the Chevies don't built-in X is worse!'.
You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.
That sounds like a win-win to me.
>You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.
I find it interesting that, of these two systems, it is Volvo's widely-deployed system that is described as 'arbitrary'. If this argument had any merit, it would be unsafe for this system to be on the road, 'conflicting' with the car's human driver. On the contrary, the statistics clearly show that these systems reduce the frequency and severity of accidents.
Maybe it makes sense to rely on arbitrary vehicle specific fallbacks, but long-term it doesn't as I'm sure They don't want to be dependent on a specific model. When you request a ride, do you want to care about whether your getting a Honda or a Volvo?
... is called redundancy and is the basis of building mission-critical, safe systems.
Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]:
>For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted
That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said.
In fact, there's yet another different, redundant, potentially conflicting safety system: the human safety driver.
If only one were to trigger, it still wouldn't be a conflict.
And yeah, redundancy through using non-identical systems is very much a thing, and considered much safer than having two identical things (which might make the same mistakes).
Source: I've worked on ASIL D safety-critical automotive systems, where diverse redundant subsystems were actually a design criterion.
There are trade-offs here, sometimes we might decide to take a very small risk, after weighing it up. A famous example is ETOPS. Unlike propeller engines, jet engines for airliners are ridiculously reliable once running. ETOPS (jokingly "Engines Turn Or Passengers Swim") is a scheme which over the past few decades has gradually allowed twin engine jets (which obviously just become over-engineered gliders if both their engines fail) to fly over oceans and other large distances with no airports, on the rationale that although total engine failure would probably be deadly without anywhere to put the plane down, two unrelated sequential engine failures are now so unlikely as to be overall negligible risk to the passengers.
But that needs a risk we understand. If you have no idea what the risk is, you can't make that trade, so redundantly safe systems are the obvious choice.
If one system detects an obstacle, stop the car. Even if this system was failing and the other wasn't detecting anything.
And if one system fail, yes the car should stop if it's not safe to drive with only one of them.
Either your eyes or ears may fail you now and again, but you're better off with both.
No?
I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong.
Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.
Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.
Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action!
If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening.
The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.
Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.
No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies.
Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.
https://www.youtube.com/watch?v=WbW2UgmjJUA
IIRC it was discovered this happened because the owner hadn't purchased the "Pedestrian Avoidance Package".
And shortly before that, a Volvo plowed into a parked truck (in a demo):
This was "was the result of an earlier flat battery that temporarily disabled the system due to low voltage" [0]
The pattern here is not only due to the collision systems themselves, but to the system around the collision avoidance system.
[0] https://jalopnik.com/5648126/volvo-pedestrian-avoidance-cras...
I'll sound harsh, but to be honest it seems that Elaine Herzberg's behaviour also played a role in this drama.
The video is a perfect example of jaywalking.
Dressed in black, crossing the street at night in a dark place, no lights, not reflective jacket or zebra crossing area, not paying enough attention to traffic. In the video she's not even looking at the car. There is not signs of the woman stopping or changing their trajectory to avoid the crash.
This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here.
From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way.
Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did.
The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy.
It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on.
I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.
https://news.ycombinator.com/item?id=16658566
Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in).
The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely.
The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct?
The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.
Multiple redundant systems that do not need to talk to each other to coordinate is generally safest.
And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?
The point is that Uber should only be disabling these safety features on public roads if their technology is superior.
If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks.
Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.
Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.
It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.
If you had a firewall in your network, for example, that had both a firewall system that you configured, and alongside it, a firewall configured by a third party with "best practices" in mind, that would be frustrating to configure and test because you'd never know for sure if it was a configuration you didn't setup correctly, or if it was this other firewall that you couldn't operate interfering with whatever it was you were doing.
When designing things like this, step 1 is you remove variables that could change the behavior of the device in question without you knowing why it did. I'm sure once completed, a self driving car system could well be made to interface with the safety systems otherwise present, or could be designed in such a way where they wouldn't conflict. When you're in the thick of building that system, however, an additional (and by your comment, not exactly predictable) system to contend with, that you have no access to, is going to be a pain in the rear.
Again, the stock auto break systems rarely engage unless there is real danger. By that I mean last second breaking, absolute last second. Most people can drive a lifetime without ever having them engaged. If you cant program a self driving car that can handle that then you're in the wrong business.
If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.
How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.
I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).
It seems like "not running into stuff" is about the only level of coordination necessary here.
I have no idea how this input would be interpreted by an autonomous driving system. However if I was designing such a system I would be pretty concerned about the potential safety impact. At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation because Volvo’s system is probably closed source and likely exists as a hardware black box. At the end of the day I would probably be more comfortable designing my own collision avoidance system than relying on one that I couldn’t test easily, and that might spit unexpected inputs into my already-complex system.
So no, I don’t see this is a simple question of “adding redundancy”. I think what Uber did here probably makes good sense. The fact that their own collision-avoidance systems didn’t fire is, however, a problem.
Also the collision-detection systems should be getting tested so that they improve too. You want your fall back systems to be more reliable than the regular system. So if you turn off the collision-detection systems never fix the false positives and false negatives then you'll always have these problems.
> At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation
Why? Does Volvos system raise events differently in context? I would imagine it's pretty simple to simulate an event by driving the car into a wall and capturing a dump of the CAN bus.
The autonomous system could take the Volvo system firing as an event like anything else that it needs to reconcile in the system. If anything the autonomous system could be used to improve Volvo's system by capturing extensive data dumps of the events leading up to misfire events.
But -- here's the crux of the thing -- it's not a new goal, it's the same goal they (should) already have.
A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes.
And this second category of code is something that already exists, deployed in production today by multiple car manufacturers.
That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.
Exactly the same deal if there's an AI driver.
The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong.
So basically, the driver (be either human or another system) did something wrong already if that system kicks in
That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it...
I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc
This way there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.
Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots.
I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!)
Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them.
I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.
Aerospace (and medical) is where things are much better.
In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.
But compare with the other discussion currently on the front page, where users point out system subsumption is a basic safety principle. You need a safety system to fall back on if the AI fails (as it will, at least at this stage). It seems grossly negligent if Uber didn’t re-implement collision avoidance at a lower level.
Edit: link to discussion I’m thinking of https://news.ycombinator.com/item?id=16681611
Getting information that some automated system initiated some operation is almost certainly harder. System only knows that the car is not behaving as expected, but it doesn't know what caused it. You could probably figure out how to determine if something like collision avoidance was triggered (large shifts in expected vs actual outcomes), but in case of something like ABS you could end up with two systems fighting for the control over brakes in emergency situation.
If the switch state changes from closed to open, [driver|AEB] have intervened and Uber's system should relinquish control of that sub-system.
No, both cases are fundamentally different. If a human driver has intervened, the system should expect the human wants to take over all subsystems: braking, steering, accelerating. If the AEB has intervened, the system should expect it will take over only the braking, and only for a limited amount of time; it shouldn't relinquish control (and doesn't have to, since a properly-designed AEB will be an override of the braking control), and should be ready to take over immediately after it releases (and probably bring the car to a safe stop as soon as it has control back).
That is, the self-driving system must know whether the command came from the human driver, or from another system.
I'm not convinced the two should be linked in any way, human driver or AEB or whatever else. If the human is taking evasive action, the auto-drive is getting input from all the available sensors.
This shouldn't be that hard (relative to the other challenges being faced).
Compared to all the other aspects of making a self-driving car, which are genuinely very, very hard and require groundbreaking technology, it seems incredibly lazy and negligent not to get this bit right.
The auto braking system doesn't do steering, nor indicators. So, you can't treat those the same.
Then a lot more testing data could be gathered by outfitting random vehicles (taxis, etc) with the tech, and analyze (and further refine the software) around every event where the software and human driver differed in opinion (i.e., whenever the vehicle abruptly changes speed, did the software detect that it should have hit the brakes at / before the same time).
Both Volvo and Uber could have opted to use this car instead. Even choosing a SUV is question-worthy, as they're known to be more dangerous for pedestrians due to higher bumper heights: https://en.wikipedia.org/wiki/Criticism_of_sport_utility_veh...
IMO there is at least some negligence on their part for not choosing a car that is more likely to protect pedestrians.
The V40 is also a rather old car and is based on an old platform. The XC90 on the other hand was their newest car at the time the Uber deal was made and is based on their latest platform. So it is not unusual that both Uber and Volvo would prefer the XC90 over the V40. Besides, given the newer technology in the XC90 it is quite possible it is better in pedestrian accidents than the V40 (with the safety systems enabled on both).
Like the Intel statement from article. Hope there will be some sort of requirements for this equipment defined. Or technically benchmarked before allowed on road.
> Intel Corp.’s Mobileye, which makes chips and sensors used in collision-avoidance systems and is a supplier to Aptiv, said Monday that it tested its own software after the crash by playing a video of the Uber incident on a television monitor. Mobileye said it was able to detect Herzberg one second before impact in its internal tests, despite the poor second-hand quality of the video relative to a direct connection to cameras equipped to the car.
https://www.google.com/search?q=mobileye+sends+tesla+a+lette...
This came about because Mobileye was honest about their capabilities: https://electrek.co/2016/07/01/tesla-autopilot-mobileye-fata...
Tesla, in their immediate response (same article), basically tried to imply they have magic abilities to do much better, which, based on later crashes, we know is not true.
Mobileye, in response, chose to end their partnership with Tesla (and yes, it was Mobileye who did it. The letter they sent ending the contract can be found if you browse around, after Tesla explicitly asked them to continue and signed a multi-year contract earlier in 2016)
If you have an automatic but a brutal, on-off style emergency-braking module based on, for example, lidar then it's much easier to develop another, maybe less aggressive but much smarter, auto-braking module based on the regular camera inputs. You can count on the failsafe system to act if higher-level modules fail. With braking in particular, maybe even a couple of redundant low-level emergency brake systems would be a good idea before you even consider any of the more high-level systems. You really don't want to hit jaywalking pedestrians or wandering drunks because both kinds do exist in reality.
Similarly, once the car has a couple of obstacle-detection and auto-braking systems up and running, it's easier to work on autonomous driving because you can be confident that your computer won't be able to drive the car into anything as these isolated and separate systems will take care to stop the car.
Consequently, when you're developing the navigation and routing system it has to be able to rely on the driving functions that deals with the present-time traffic situations, like keeping up with the flow of traffic, slowing down, changing lanes, etc. Again, same thing as with emergency brakes: to develop a smarter higher-level system you need to have lower-level failsafes in place.
In the case of autonomous driving a failure should mean a harsh but unnecessary full stop or the inability to continue, not a collision or the inability to stop. They should definitely have left Volvo's own system on as an additional failsafe mechanism.
Volvo is the gentleman here, with its “can’t speculate on the cause of the incident” statement.
Simply taking a crappy dashcam video with low dynamic range suffices; it will make an illusion that anything not directly in the headlights of the vehicle is invisible.
I don't know how people can believe a single word of the early official statements which claimed that the victim "suddenly" appeared in front of the vehicle. The problem of "it's too dark to drive" has been solved a hundred years ago with the invention of headlights.
Here's an experiment anyone can try: turn on the headlights on a dark road, and see if you really see nothing in the lane to your left (driver side). Whatever you do see (which, under normal circumstances would be at least three seconds worth of road ahead) is the way in which the released video is misleading.
The speed limit is the max limit, conditions permitting.
But it also seems like "It works better than the stock production autonomous braking" would be a gating factor in putting these vehicles on the road.
Low visibility on a clear day is simply lack of illumination, and it is a horrible excuse, because:
1. The car is equipped with headlights, which solve the "limited night time visibility" problem;
2. The car is equipped with multiple types of high-quality cameras;
3. The car is equipped with a LIDAR, which is an active sensing system.
Finally, if one accepts that the visibility was limited to 50 feet, as the video misleadingly suggests, THEN one has to concede that the car was going too fast for how much it could see.
People are taught to drive so that at least three seconds worth of road ahead can be seen. If you believe that either the car or the driver were seeing the equivalent of what the webcam footage was showing, the car was going unsafely fast.
Hanlon's razor would suggest, though, that the car simply wasn't able to process the data available to it. We'll know for sure if the investigation is successful.