Former Uber Backup Driver: 'We Saw This Coming'
citylab.com
citylab.com
The only viable autonomous cars are ones that will never require human intervention, because humans aren't able to intervene effectively and may make situations worse as they snap out of a distracted state. But we can't get to full autonomy without first testing the technology, and no artificial test conditions could replicate the complexity of the real world. Society will either learn to accept some avoidable deaths during the testing phase or will ban the technology. I suspect a few more incidents will result in a ban, and we will not have fully autonomous cars for a very long time.
The human side of automation [0] is an issue that has been known and written about since at least 1983 by Lisanne Bainbridge and Don Norman in 1990.
For example, imagine a self driving truck that works perfectly, but only during the day, on highways, when it is not raining.
Such a system would still put millions out of work, as firing half of the truck drivers is possible if something works when it is sunny, on the highway.
What we really mean when we say a system is reliable/perfect is that the conditions which can cause failure are so rare that they can be neglected. Any system that depends on things that are inside normal experience: rain, day/night, errant pedestrians, earthquakes, and so on can never be trusted to act reliably/perfectly. A "reliable" system might be excused for messing up under conditions such as meteor strikes or falling space stations.
I think we can conclude, absent conditions of wilful neglect, that this hypothesis is false. I thought this, too. But people are reacting reasonably to Tesla and Uber’s, erm, faux pas.
Imagine if instead of saying the word "perfectly" I said "works better than humans".
Do you reject the idea that there are situations, such as sunny highways, that are much much easier than everything else? And that if we can solve this much easier problem, it would still be world changing?
Why couldnt that be trusted to work better than humans? It is a much easier problem.
One person's pedant is another person's clear conscience. There's massive disagreement in the self-driving debate as to the extent to which we are dealing with safety critical systems. As you can no doubt tell, I'm firmly on the safety critical side.
I've a fair bit of experience with designing and shipping critical infrastructure for emergency services. One has to ship, but in my experience the majority of people don't have the necessary eye for detail to make a safe system. This is why we have process: to allow fallible people to ship safe systems (and to avoid delusion in the minority who do have an eye for detail). The self-driving field is still developing their processes, so they have to be conservative with their shipping.
Many of those arguing for self-driving are under estimating the safety of a typical human driver. The bar that a self-driving system has to meet is very high. If conditions are easy for a self-driving car then they are probably easy and safe for a human as well.
Accidents are skewed towards a subset of poor conditions involving speed, alcohol and fatigue. Is it fair to replace a safe driver that avoids speed, alcohol and fatigue with a system that whilst safer than average is still less safe than the specific circumstance?
Maybe an interim use of self-driving technology is to have the person driving, and the self-driving system in a co-driver/critique mode acting as an "unsafe driving detector"?
this is our current state of the art with automation and safety and yet we allow companies to slap a radar and a pretend safety driver and hurl a ton of metal down the pubblic roads
autonomous vehicle will undoubtedly at some point be safer than human, but shortcutting known limitations like radars vs dodging static objects because 'drivers are the backup' is lunacy and the recent deaths are just a symptom of people never having worked in safety system design playing fast and loose with human lifes, and the companies which enable that should be made an example of by courts.
some years ago serious automaker did their analisys and come up with the concept of designing the anticollision system first and use those as a backup to autonomous driving technology, that somehow has been lost in the race for profitability.
This is what bugs me most: Who on earth ever thought it's a good idea to allow a company like Uber with their history of corner cutting and shady business to test such dangerous technology on public roads?
The mind boggles.
Or imagine if you just look up the weather ahead of time and only run the self driving trucks if there is a less than 5% chance of rain.
These situations are still very common. You could replace all trucks on the highway in any desert climate, for example. And that's still many billions of dollars.
What you're imagining is a car on a fixed track - a railcar. We already have trains but the US does need to build out their railway, yeah.
There are lots of places where it is sunny, and not raining and it is daytime.
Just don't run the trucks on self driving mode during the other times where they don't work, and we could still save up to 50% in labor costs.
Obviously, though, the market for self driving taxis is different than for self driving trucks, but once again, we could just only use them for trucks and still save Trillions.
You mean pedantry :). -- Admiral Pedant.
I’ve done quite a bit of inventory manual work during college/university Holliday’s and even with such a limited experience I can’t even begin to explain how wrong you are because it would go on for hours.
In a neighbor country wich is pretty much considered as a leader in industrial automation (Germany) it’s almost mandatory for engineers to experience months of low level jobs to get a grasp of what practical real world problems are when working.
This is pure Silicon Valley overconfidence. Software "move fast and break things" approach really don’t translate well for hardware. And in this case I think Uber will learn this lesson the hard way.
All I said was that the act of loading/unloading etc. would be done by different people than the truck driver.
Also, we already have semi-automated warehouses, it is not unimaginable that the drones driving around with boxes could be adapted to able to help load a truck. Probably with the help of another machine. Not saying that this process needs to be void of any human at all - just that the truck driver wouldn't be part of it.
I really didn't say that the inventory work would just disappear - it would just be moved from the truck driver.
Secondly truck driver usually are in charge of the final delivery to shop or customer and have to deliver good to buildings that are not by any mean standardized.
Having a human on board that know how to operate the truck and how to properly unload it is a feature that will probably be missed out. As It would be pretty unwise IMHO to rely on external people to "not fuck up" when handling heavy objects around a costly automated truck.
So maybe I am totally wrong, but if we concentrate our current R&D on building expensive automated truck so that we can reduce cost by firing driver... Instead of focusing our R&D on producing more economic and ecological truck and alternative transportation... we might as well be fucked up as a specie.
> As It would be pretty unwise IMHO to rely on external people to "not fuck up" when handling heavy objects around a costly automated truck.
I agree that this will be an issue and the people on the ground would need to be educated. But I hope you don't think this problem is unsolvable in practice. We somehow do it for aircrafts (I don't think the pilot does the actual leg work there).
Again that’s nor simple neither cost effective by any mean. Your are dealing with a lot of externalities here.
Logistics is a chain, you can’t replace a link and expect that the others will magically adapt to the new element. It’s like an API you can’t expect users to be happy if you break all compatibility for the sake of progress. And again for API only software have to be adapted which is far easier than hardware.
I don’t doubt than sometimes in the future when all the caveheats will have been addressed and that some standard will have been defined yes you will have automated trucks. But it’s not simple and I don’t see them comming before at least 15 years and in any case after passengers automated cars.
Perhaps the first to adopt this would be postal services and large stores with many outlets throughout a country with their own trucks that do the same route every day. They have control over each destination and it would most likely be easier work alongside existing logistics and everything is kept within the same company.
And sorry if I’ve being boring it’s a deformation from Geography my first scholarship. Geographers tends to focus on systemic interaction rather than on individual components.
LOL. The other day I was in Mexico on a cave diving trip. It rained, short sudden downpour, out of nowhere. The carnage on the highway just in the few miles from the dive shop to the cenote was insane. No fair-weather robot would stand a chance in those conditions.
Actually, I think a test environment could be a lot more complex.
It's mind blowing to me that there doesn't seem to be some big track somewhere with automated crash test dummies riding their bikes across the street in the dark. Automated cars running red lights and stop signs. Automated accidents in front of self driving cars. Automated kids chasing a football across the road.
It's mind blowing to me that governments do not require self driving cars do well on all these test before they allow the cars on the road. Governments should prepare these test, and car technology should complete the tests before they are allowed on the road.
And when a car fails any real world situation, we can step back and create new tests.
Really: We can test to any complexity we want. A plan lading on the road? A flood? A heard of sheep or flock of birds? A dust storm or other weather than might prevent the cars sensors working.
http://www.businessinsider.com/ubers-fake-city-pittsburgh-se...
Do they have crash test dummy kids running out on the road? To they have road work with giant potholes?
I can't even imagine the amount of data being generated by the fleets at waymo and others but compared to all the infinite possibilities I must assume their data set is tiny.
Well, there you go.
Normally it's fine to test by using models of the world rather than the real thing. But in the case of self-driving cars, driving conditions are too complex to simulate and people could die if things go wrong. My point is that we are left with a problem where the only viable test case is the real world, but it's likely that after a few more incidents people will favor banning self-driving cars rather than letting them improve at the cost of human lives.
At the very least a test track should be able to teach the AI how to react to a cyclist crossing the road unexpectedly.
We'll never reach zero risk and you can always contrive some scenarios where automated systems wouldn't help. But it's still worth pursuing incremental improvements. For example, street sign recognition is getting pretty good. In a few years it should be possible to build a system that detects when the driver is about to run through a stop light or sign, and automatically brake.
Fundamentally that is what's going on here. This is a massive power grab. This is convincing people to give up control of their transportation under a facade of improved safety and convenience.
Imagine streets full of powerful multi-ton wheeled drones relatively few people have control over the software operating.
Ignoring conspiracy scenarios, just imagine how 0-day hacks of that system manifest.
None of these situations are possible until you convince everyone to take ownership of one of these robots, or at least accept and utilize a ridesharing fleet of them.
In the meantime level 5 auto automation can mature in test areas.
This is not a safe option, particularly on certain roadways. Accidents increase when you have a higher levels of relative speed difference between lanes of traffic. Having "turtle mode AI cars" isn't going to help us.
Speaking of the elderly.. they often end up in fatal accidents even at low speeds. Their frail bodies combined with strong safety restraints are not a good combination; particularly in the rear seats where their level of risk increases to nearly 2x the front seats.
Otherwise I agree with you.
It will require an effective "early warning system" which foresees issues it may not be able to process --however, this will require more inter-automobile communication and extravehicular communication (GPS, other inputs). Relying on only what the vehicle has on-board will probably not be good enough, for now.
These supporting systems would communicate "trouble ahead" (ambiguous information, incomplete information, lack of information, unknown information, etc.)
This is essentially human augmentation. The regular collision detection in many cars now is a good example. It's a relatively simple system that should stop a distracted driver from crashing.
Musk said something similar for the Tesla trucks, the auto-pilot can take over if the driver falls asleep or becomes unwell. Again this is augmentation.
The only way to keep the driver alert is to have them driving all the time.
This however, can't be sold as 'almost self-driving' to a gullible public.
This contributed significantly to the Air France 447 crash in 2009.
see: https://99percentinvisible.org/episode/children-of-the-magen...
and more broadly https://www.youtube.com/watch?v=pN41LvuSz10
They're 'set' in the sense that if communications is lost between the pilot and ATC, there's a pre-determined and more importantly pre-authorized path for the aircraft to fly through.
> A plane on autopilot would never have to dodge a pedestrian
It happens quite a bit, which is why we invented TCAS. [1] A common misconception is that _all_ traffic is required to file a flight plan, and that's only true for IFR (instrument) traffic and not for VFR (visual) traffic. So, an IFR flight can, and will, encouter VFR flights that interfere with their flight.
> and would be separated from other planes by thousands of feet.
In certain situations, in transit regions and during clearance/delivery, this is not true.
1: https://en.wikipedia.org/wiki/Traffic_collision_avoidance_sy...
https://flightsafety.org/runway-incursions-down/
But realistically it's not like all flights could be done completely on autopilot. It's not uncommon for a flight plan to change due to delays at an airport.
But Uber never has been and never will be trustworthy. It's a toxic organization that will continue moving fast and breaking things - sometimes intentionally - until it's stopped by either the market or the government.
When driven in a small number of states that have beneficial weather conditions. I don't think we've done _nearly_ enough testing to come to this conclusion yet.
Genuinely curious if you've come across accident statistics that break down human miles in to similar categories as the kind of miles autonomous cars drive.
Edit: I’d love to hear a counterpoint, although of course I accept that’s not required. Like the discussion about fusion, people seem to argue from a point in the future when all of the problems and limitations of today are gone. Let’s try arguing from what’s possible now instead.
https://theconversation.com/the-trolley-dilemma-would-you-ki...
The type of mentality that would sacrifice one life to save five is instinctively assumed by a lot of people to be the sort of mentality that would deceive themselves and/or others about the correct statistics and the uncertainty thereof. Assuming the ratio of 1:5 to be accurate and worthwhile is largely missing the real issue, in my opinion.
I understand the computer science concept to abstract the term but when you're talking to the drivers of the cars this is in retrospect an awful term.
Rather depressingly all I can think of is the ED 2000 from Robocop that shoots citizens who don't comply.
The Caltrain calls incidents where it hits a pedestrian "tresspasser strikes". Always thought this was awful.
> In it, they are trained to keep hands hovering near the wheel at all times so that they can quickly take control when the car does not safely respond to dangerous road conditions or “noncompliant actors,” such as people walking in the roadway
(They don't do any pointing though, because that would require taking their hands off the wheel)
A certain amount of over familiarity with western driving conditions blinds these attempts to the reality that there are two worlds on the roads - the legal and socially manufactured layer and reality.
Which is why looking at the actual problem space, like nations where road rules tend to be ignored, makes it clear that what you actually have to solve for.
It has been highlighted before - but the autonomous car can only work if the road itself is active / aware
Development of these changes will be spurred once self-driving cars can make more efficient use of the road; the immense pressure of society to get rid of traffic jams will make sure of that.
Maybe some nations won't implement these things nationwide with 100% coverage, but they certainly will implement changes for certain roads. Highway roads first, then collector roads, and lastly neighborhood roads.
Taxes typically taken in at the gas pump are being skirted around by having electric cars. Do we just start charging MUCH more for registration? Especially cause these cars have the potential to operate most of the day (except when charging) so they will be causing more wear than a normal car.
http://www.pavementinteractive.org/equivalent-single-axle-lo...
An 18-wheeler causes 10,000 times the wear of a car, approximately.
I'm not sure what you are referring to when you say electric cars have the potential to operate most of the day. How is it that gasoline cars don't?
Presumably, you would pay the RUC in the same manner for electric vehicles at some point, if it isn't already in place. (I don't own one so don't know if they're already required.)
I'm unsure of how this change will get traction. I suspect states will eventually explain their infra budget cuts.
There will likely be a bias in the future for incidents like this to show some level of driver distraction. This makes it easy to blame the driver when the idea of a human as a supervisor is fundimentaly flawed.
>Both Kelley and the former driver in Tempe were dismissed from their jobs with Uber earlier this year for safety infractions: Kelley said he was let go after rolling through a stop sign while he was operating the car, which he disputes; the individual in Tempe said he was dismissed for using his phone while the vehicle was in motion.
The bit about level 3 considered harmful makes a lot of sense and isn't something that I would have intuitively thought of
What makes you so confident that the mere two years of running the program is enough to reliably calculate that number, when drivers like the one in the article managed to last more than a year until they got fired?
The problem in Tempe was that the car did not recognise the pedestrian, so how could it be relied upon to warn the driver? Perhaps a better option would be to buzz when the driver is not looking at the road, as was the case in the Tempe accident. This technology is already in use [0].
[0] https://en.wikipedia.org/wiki/Driver_drowsiness_detection
I imagine it isn't cheap though to add and even if it was reasonably priced I imagine they would make it optional in case people didn't want to add it to their car price in which case many would simply say "Nah I don't need that. I'm a great driver!"
The car has the right of way, it didn't consider she might enter the space anyway.
For these specific vehicles shouldn't they have some hand placement sensors or wifi strength detectors or mobile signal detectors or eyeball monitoring at a minimum?
Or use a custom app to "lock" the phone while driving, exit app and the car pulls over.
Not that the driver would have prevented this incident, but seems like a logical requirement.
So to me the question is: What is making this difference? Is it the level of training for the instructor? Human interaction? Is it the sign on their car that shows it's a lesson car to the other drivers? Or is it something else?
PS, this post is based on the way driving lessons happen in the Netherlands. I'm not sure if it's completely comparable to the US.
What is making this difference?
These zero-experience students need the instructor to intervene from time to time, thus justifying and reinforcing the instructor's habit of vigilance.
On the other hand, if an autonomous car has one near-accident every 100,000 miles, it's not often enough to maintain and keep strong a habit of vigilance.
Note that we already do this to keep people vigilant in rote tasks, e.g. for airport x-ray scanner operators or train drivers.
Maybe part of the problem is that the self-driving companies view their drivers as temporary measures, so choose to invest their resources into the machine component rather than the human component?
They are also probably expecting a student to make mistakes, which puts them in a different mental state than with a car that is right almost all of the time.
and have you seen the vide of the "instructor" during the fatal uber crash? completely zoned out compared to what a human-instructing instructor would do.
- You want the driver to be able to call 911 if they're involved in an accident, or are the victim of a crime, or have a medical emergency.
Presumably, they'd only be firing people for using their phones while the car was driving itself, and they were supposed to be supervising it. It would be OK to use a phone while the car wasn't moving.