New reports suggest limits to autonomous vehicle performance
spectrum.ieee.org
spectrum.ieee.org
TLDR: Cruise had, what some would consider, to be a disengagement incident but didn't report it because it does not fall into the categories defined by California.
The core issue here is that the disengagement reports should not be the end-all-be-all of where companies are. As the ieee article calls out, we have no clue what kind of testing the companies are doing (how hard they are pushing their systems). There is also some room for companies to avoid reporting certain types of incidents.
Waymo did 2 million miles in 2017, but only 330k were in CA. We are only seeing a slice of their data.
If we (the public) want a better picture of where companies are on this, the rules around what is reported need to be changed. They possibly need to also be done at the federal level so no company can hide their public testing in states that don't require reporting.
Disengagements is a useful number, and the disengagement numbers say so much more about the real state of self-driving cars: That they fail as often as a non-synthetic oil change. And if humans had driving system failures that often, we would not be considered good at driving.
The longest number a company should be marketing is how many miles they've driven between disengagements. Like the "days since an accident" sign at a workplace: Fail, and you go back to zero. The numbers will be smaller, but they'll be a lot more meaningful.
Risk also skews results: Pilots taking off and landing on beaches in Alaska aren't comparable to pilots taking off from airports on the basis of incidents per 1000 hours.
i'm not sure what benefit there would be to gaming the disengagement stats. If anything, i think the incentive would be the other way around - inflate your number of disengagements so your competition thinks you're further behind than you actually are.
The point of testing is to bring a product to market, and if your product sucks people are going to figure that out relatively quickly. hiding your disengagements from the california DMV isn't going to do anything for you except build hype your product can't live up to.
Why? I'm as curious as anyone, but what is the argument for mandatory disclosure of disengagement numbers?
It all depends on the content of the miles. And without transparency about that it sounds impressive but could very well be 95%+ that doesn't contribute much and where your average toddler without a driving license could have handled the car.
Driving in Mexico City last month I was appreciating how intensively interpersonal driving was: it seemed all drivers were constantly computing the risk appetite of all other drivers and acting accordingly; navigation, avoiding pedestrians, and following the law (to the extent possible) was the easy part. Maybe there's a way to avoid issues by providing coordination mechanisms, but I don't see how this would work as long as there are human-driven legacy cars. So will self-driving cars need theory of mind?
You can observe similar plateaus in power systems (coal, nuclear), medicine, etc.
Self driving cars (road safety, congestion?), renewable energy (atmospheric CO2, air pollution), etc seem to fit that motivation.
The next quantum leaps here seem to be things like teleportation (for transportation), transhumanism (for "life extension"), or interplanetary colonization (for ?). All seem like very remote possibilities today.
A few examples come to mind.
There is the field of prime movers and combustion engines. Efficiencies are set by the Carnot and Rankine cycles, and are constrained by the hot/cold side ratios. For most fuel-based engines, the maximum attainable efficiency is on the order of 50%, and in practice, an ICE (Otto-cycle, Diesel, or turbine engine) tends to hit about 30% efficiency in actually use.
(Ironically, it's coal-fired power plants, with predictable power curves and high furnace temps, which reach some of the highest Carnot efficiencies, though marine powerplants, often two-cycle Diesel engines, come quite close.)
You're simply not going to get better.
Even electric motors where they rely on thermal electricity generation are faced with similar limitations, despite the electric motor's ~85% - 95% efficiency in converting electricity to motive force. It's the decoupling of motive force from energy origin which is a primary advantage.
If you look at the field of lighting, changes in fundamental techniques have resulted in a tremendous increase in lumens per unit of input energy. Open fires, torches, and oil lamps relied on blackbody emissions (mostly from soot in the flame and air) for light, as did incandescent electric lamps, through from a heated filament. Fluorescent bulbs rely on specific chemical properties and emission, and LEDs on quantum effects. It's hard to imagine much of an increase beyond this, though there might be possible avenues.
If you look at automobiles and aircraft, initial innovation as measured by patent filings peaked in the 1920s, only a couple of decades after initial mass-market offerings. Much of what followed were changes in fuels (from NGLs to petrol/gasoline, the disaster that was leaded fuel), standardisation (placement of controls, signalling lights, wipers, components), safety, handling and controls, emissions controls, and other dialing in of performance, reliability, etc.
Rober J. Gordon in The Rise and Fall of American Growth makes the case that the DC-3, still in active commercial use was the pinnacle of aircraft perfection, attained in 1937, only 24 years after the Wright's first powered flight. Principle changes have been jet aircraft (after 1945), the jet widebody (707, 1958), and in the past decade or so, materials developments finally moving on, slightly, from the airframe designs of the 1950s.
Consider that the U.S. heavy bomber fleet of B-52 aircraft will be flying not only the same design as first flew in 1952, but the same aircraft, last manufactured in 1962, through at least 2040, some 88 years after first manufacture, and meaning that the aircraft will be at least 78 years old.
There's some room for improvement of the aircraft, but often the benefits simply aren't worth the price. That was the case for an engine retrofit programme explored in the 1970s, which determined that the fuel savings would not be cost-effective. (It's possible that subsequent petroleum economics have invalidated that assessment, though I'm not sure.)
There's also a role played by materials and the properties they provide: organics, stone, ceramics, metals, fibres, synthetics, composites, and more. There is an opportunity frontier of properties, side effects to consider, abundance and cost, and generally constraints on what can be provided. You'll see this in vehicles (engines, frames), devices, energy storage (especially batteries), fabrics and clothing (much of the "look" of 20th century decades is dependent on what new fabrics became available), etc.
In the information field, you have a fundamentally different dynamic in that capabilities scale superlinearly with chip density. That is, as you pack components ever close, the cost (and energy) fall by factors of two, about every 18 - 24 months according to Moore's Law. What ends up happening is that whilst, yes, former levels of capability are now exceedingly cheap to provide, additional, previously unattainable levels of capability are also possible. So that where Nokai had perfected the "deliver voice and SMS in my pocket" capability, the iPhone and Android upped the ante with music, Web, email, expanded messaging capabilities, and apps.
And surveillance, privacy invasion, attention erosion, exploding batteries, and other negative affordances.
There's a curious point that in technology the market seems to be far less about "how much would you pay for X" and far more about "How much X can you give me for Y dollars?". That is, price points seem fairly constant whilst capabilities expand. It suggests we have a budget for capabilities, and that manufacturers aren't interested in investing in providing abundant, but low-priced devices (at least not in advanced markets).
Summary: If it moves, delivers material or energy, or otherwise interacts with its environment, expect a fairly early plateau. If the device is informational you can see much longer periods of evolution, but also some constancy to other elements of the technology, particularly around cost. The Jevons Paradox means that as costs of providing capabilities falls, induced demand increases (especially in transport, comms, information, data). (A Gresham's Law dynamic also means that simpler / more common uses dominate as markets grow.) Physical limits still constrain, but may be expressed in unintuitive ways.
Where's my automatic lawn mower? Where's my robot to unload the dishwasher and put everything away?
We need to find other consumer technology to use AI in before we risk our lives with self driving cars.
That's how we got our century of development. Testing, verifying, refinement. It wasn't by pushing out new stuff to a batch of users and seeing if everything worked out allright.
Other than that, Mrs. Lincoln, how was the play?
Just recently it was discovered a series of air bags manufactured in a small window of a couple of days, would instead of protecting the occupants, would emit high-speed shrapnel into their chest and face killing them. Quite an oversight.
If your airbag fires you’re already in an accident and might die anyway. You’ve already accepted that risk. There are probably better recall examples but I’m not sure it’d disprove the underlying point though.
If the automakers can't catch problems in a simple system like this, just imagine what sort of new and exciting failures we'll see in an inadequately-tested car autopilot! These things need to be very thoroughly tested before we allow lots of them on the road, and that needs to happen before they omit the steering wheel.
[1] http://www.automotive-fleet.com/channel/safety-accident-mana...
[2] https://www.autoblog.com/2015/10/30/fca-recalls-894k-total-v...
Switching from a human driver to software automated driving is a much bigger change in automobiles than has ever come before. It's also one with a much larger reasonable test surface, by orders of magnitude. Systemic risk? How about meltdown and spectre vulnerabilities? The failure mode of most new hardware in cars requires either edge cases or corner cases to result in the worst case scenario of a crash at speed. And yet that's a very common failure mode in autonomous driving. You can't realistically have a car that can autonomously drive itself 90% or even 99% of the time, it must be nearly exactly 100% of the time to be worthwhile and safe.
(I'm referring to devices/products such as the Roomba.)
Cars make sharp turns and get into tight spaces on unmarked, uneven surfaces, with pedestrians walking everywhere around the car.
That's what a self-driving car needs to support. All road rules out of the window, tiny roads you canbarely navigate, and due to a festival everything is redirected around and all your maps are useless, you're told to drive a short bit through a park because there's a parade on the road, the only way to tell where you can and can't drive being a nonstandard sign and a police officer telling you.
This is a real life use case of driving, as given in many European towns. For additional difficulty, do a small Weihnachtsmarkt in a rural place, where you'll have the same, but with so much snow you can barely see anything.
There is a physical challenge arising from the form factor, such that your intelligence won't help. Those robots don't do stairs and tight places very well simply because the hardware is that way.
In driving, the robot has the same controls as the human. It can turn the wheels, brake, apply gas. That basic playing field is level.
It's all about how well you make the AI see and understand, and translate that into turning the wheel, changing forward/reverse, and hitting gas/brakes.
Robovacuums seem to have hit a sweet spot - they do just enough on their own to justify their costs by saving you time and effort. Solving the corner cases you mentioned is possible, but there is not a strong enough economic incentive for that.
I know enough people that drive terribly to see the obvious benefits to autonomous driving. It doesn't need to be perfect to have a net-positive effect on road safety, it just needs to be better than a human on average. And based on the evidence, it seems they're doing a damn good job of achieving that goal.
Google can't even make GPS mapping work perfectly - it's apparent if you use it for a day. Not to mention that AI vision systems are susceptible to adversarial attacks, which means they can be hacked by external images. This paper just came out yesterday: "Breaking 7/8 of the ICLR 2018 adversarial example defenses" (https://arxiv.org/abs/1802.00420). The situation is dire.
I think they are afraid that drawing a couple of lines with a marker on a street sign might make it invisible, or turn it into another sign. They can't defend against such attacks because the neural net doesn't actually understand the world, it just observes patterns. Understanding requires causal reasoning.
That's like saying a human driver can be blinded with a bright light so the situation is dire.
New world, old rules. You'll still go to jail for manslaughter (or whatever the charge would be).
Adversarial attacks are concerning for any AI model, but autonomous vehicles aren't all AI. Lots of rules and heuristics and pre-knowledge are utilized. In the end if something is uncertain it generally amounts to "slam on the brakes" which is more a ux problem than a safety problem.
Mapping the world is hard because it is complex and evolve way faster than a Google Car army can patrol it for a reasonable cost. More often than not OpenStreetMap which is a community project is more up to date than commercial map, but it’s not precise enough. Beside nobody would bet there life on something that is Wikipedia-like editable by ill-intentioned trolls.
This is precisely why all autonomous car rely heavily on computer vision and why adversial attacks should be carefully considered.
Uncertainty will indeed « slam on brake » but adversial attack don’t aim to generate uncertainty the aim to generate false certitudes.
It's true these maps don't exist for the whole world yet but it's also why all sdc companies are launching in metros. They're building them.
Vision is good at recognition but bad at localization. This is the real problem. In robotics a fundamental problem is SLAM. You should read about it.
https://www.husqvarna.com/au/products/robotic-lawn-mowers/au...
How could these cars reach Level 5 when they've only been tested for like a year or two on some roads? And I don't believe there is good enough simulation for them right now either. Even Google's CAPTCHA was asking me recently to pick the "bridge" from the pictures. There was no bridge.
My non-autonomous Honda Odyssey has cameras on the passenger mirror and rear hatch -- both of which were useless within 90 minutes of traveling from Lake Placid back home thanks to road spray and salt.
IMO Level 5 is viable for some relatively limited use cases, or with roads that have embedded telemetry. I'll believe it when intercity truck fleets convert for a few years.
- pick me up in the parking lot
- drive out of parking garage and meet me there
- trip from LA to SF etc
Its a hard problem being solved by incremental improvements — first you gotta make LIDAR work in concept (which is basically done), then you have to shrink it, then commercialize it, then commoditize it. So it’ll take a decade or more to realize, but that’s the ultimate thing I think we need to get these things into the mainstream.
That's to be expected. It even says at the top of the challenge box: "If there's no match, just click next".
Negative reinforcement is reinforcement too.
You can see it reduced rate of improvement when you dig into the numbers:
2015 0.16 disengagements per 1000 miles
2016 0.13 disengagements per 1000 miles
2017 0.12 disengagements per 1000 miles
2015: ~6250
2016: ~7692 (+1442)
2017: ~8333 (+641)
That said, even reporting the same number every year could still represent progress if they keep pushing the car harder. AKA, if 2015 = highway driving in the day with nice weather, 2017 = inner city driving at night in a snowstorm the same number of disengagements would still represent a lot of progress.
On top of that most situations with disengagement have not resulted in crashes. Really it's failing to pay attention during normal driving that's most often at fault, a 'buzzing you need to pay attention now' system solves most of the problems even if the car is not driving it's self.
It turns out to be significantly easier problem than full speed road traffic. Which IMO is why google gave up on that concept demo.
I mean, you can probably design autonomous "disengage" modes- hitting the emergency blinkers and heading for the breakdown lane is the extreme default. On a lesser level, the thing could just drive like I do. If a merge is too tricky? Just keep going straight, and recalculate on the next exit.
This is helped by that fact that modern cars seem to have pretty good "Just don't run into something" sensors already, and from my own experience as a bad driver with a decent accident record, not running into other things is most of the battle.
So yeah, I could totally see autonomous cars evolving the ability to safely get themselves off the road. Of course, you're still gonna need to do that a lot less often than every three thousand miles, but you don't have to get it to zero, just around the point where normal cars break down mechanically.
Funny, I interpret that as one guaranteed accident every year. Because by the time the autopilot disengages you can be sure the driver is after being lulled to sleep for the last 8000+ miles in no position to step into the situation in time to avoid a crash.
Self driving cars however have two goals, not hit anything AND got somewhere specific. It seems likely a car would disengage if it encountered a blocked road even if it did not hit anything. In that context we may see many cases where a car stops and 'gives up' which are in no way safety related and have no real safty risks.
While we don't nessisarily know the specifics, people have been testing these self driving cars for a while and yet crashing seems very rare even with unexpected handoffs.
It should remain in control until commanded to disengage. After all, blocked roads will be freed at some point and even if they don't it is the AI's problem to get itself out of any trouble that it got itself into. And even if a blocked road might seem to be free of safety risks that doesn't mean you can abandon the car there, you're supposed to stay in control of the vehicle until it is parked.
The only situation that I can compare disengaging with is when there is stopped traffic in the mountains and the snow moves in, that's one of those situations where you might be ordered by the authorities to abandon your vehicle and seek shelter (assuming this is possible and not less safe than staying with the car). Those situations can take many days to sort out afterwards. But in almost all other situations that normally would occur you should stay in and in control of your car, so I'd assume the same would go for an AI based system.
After that hand off someone can look at the test data and chose a better response, but caution when operating a deadly vehicle is causation is perfectly appropriate.
I feel pretty weird in the knowledge that people are operating vehicles with what to me comes across as barely out of beta software on roads shared with others.
I also don't think you can get to self driving cars without testing them on public roads. So far it's been very safe and potentially it's going to save millions of lives so I don't have a problem with this.
Further, these cars are not simply turning off in traffic they are making errors known while continuing to drive as safely as possible. That's vastly safer than suppressing errors which generally results in people ignoring them.
I'm no statistician, but I'm not sure if there are enough miles being driven that good conclusions can be drawn about whether self-driving cars are getting better or not.
Lots of testing is taking place outside of California due to other states' efforts (like changing regulations) to cater to it. So data from just California very likely doesn't tell the whole story.
Also, as has been mentioned here before, once they've solved and tested the easy cases, they might move on to solving and testing the harder ones. So there isn't an apples-to-apples comparison on test failure rates from one year to another because they could be testing different things.
From what I've heard, waymo's controller system has a long set of heuristics, many other competitors use learned algos.
Steering and speed control could be done with simple PID control. You'd have to get more sophisticated for maintaining position, but you could maintain and control position using traditional control systems. When you're using sensors you determine where you are relative to other objects and that's still within this realm.
However when certain classes of object can be expected to change position without observable indication it's hard to make a control system accurately predict that. There's no way to predict the future directly from what the car can see in all cases.
I would imagine that classifying objects and predicting which objects are likely to start moving is in the realm of ml/ai. I don't think anyone can formalize rules on that aspect of driving.
It's reminiscent of how some people once believed that a machine will never beat the top human players in chess.
1) New technology X appears.
2) Technology X develops quickly as the first 80% of gains are achieved.
3) Tech "luminaries" incorrectly extrapolate the rate of progress into a distant, glorious future; write breathless commentaries about how Technology X will eat the world in N years!
4) Technology X stagnates at the 81% progress mark, as it becomes clear that the green-field progress rate cannot be sustained.
5) Technology Y appears! Huzzah!
Perhaps the real goal is actually "level 6 - better than human" or "level 7 futuristic transport". Looking at it that way, maybe that would indeed follow Zeno's paradox (we'll never get futuristic transport) but soon surpass how a human would drive permitting integration with human drivers (being more reliable and cheaper causing disruption).
Ultimately, I think the main roadblock to automation in this realm are the vast entrenched interests (big oil, big auto) who'd lose out due to efficiency gains (just like with solar power and EVs).
[1] https://www.theverge.com/2018/1/30/16948356/waymo-google-fia...
I wonder if the flattened curve of progress in the metric described by the article (disengagement rate) is actually intentional. Maybe a decision was made to increase difficulty of driving scenarios in such a way to keep that approximately constant.
And you're also holding on to the steering wheel, which makes a big difference.
Everyone I know who has motion sickness when being in a car on a winding road never experience motion sickness when they are the driver, only the passenger.
And this could create a large business, doing many miles every day, which would be the perfect means to gradually expand the role of self-driving.
Yet,non of the big companies choose this(and i'm sure they're aware of this). Why ?
I doubt anyone is mad enough to think in the general case remotely piloting a vehicle in a safety-critical scenario using these consumer networks is an even remotely sane option
Waymo and Cruise (GM) are both building out call centres with agents that have some sort of remote control over the car.
Both Waymo and GM have emergency stop buttons connected to an air gapped computer that can take the vehicle to a minimum viable risk condition (which basically means pulling over to the side of the road) for security.
Human reaction time is very slow so to do better than a human definitely does not require zero latency.
>“The specially trained operator [then] provides a domain extension to the vehicle to help define safe operating boundaries (e.g., permission to use the opposing traffic lane where cones are demarcating a new path of travel)
https://spectrum.ieee.org/cars-that-think/transportation/sel...
Then you need to stream that data to a sort of cockpit at a distance, reliably and with minimal delay or lag spikes. Cause you know, otherwise you might kill the occupant(s).
It doesn't seem an easy problem to me, at least not if you want to make a business out of it.
Just for comparison, and maybe someone can correct me on this, aren't military drones more or less just uses for bombing and they're not supposed to run into opposition? I don't think fighters are remote controlled.
In America cars have an important cultural role as a symbol of freedom and the idea that your "autonomous" car is centrally monitored and controllable is not a very good selling point.
But yes, all "autonomous" cars will absolutely have remote control functionality of some kind, at least a kill switch.
tl;dr of the article:
- Autonomous vehicles drove fewer miles in the state of California in 2017, but maybe made up for that in miles driven elsewhere.
- The disengagement rate will probably need to improve considerably before AVs are ready for widespread deployment
- Waymo's disengagement rate barely improved year over year, but that may have been because they are placing the cars in more difficult scenarios (their blog post suggests that is indeed the case)
Maybe we're 90% of the way to practical self-driving, but won't reach 99% for the foreseeable future.
https://news.ycombinator.com/item?id=16288489
https://news.ycombinator.com/item?id=16276911
On the other hand, even those aren't holding up.