Hacker News Reader

Comment by jayjay71 | Hacker News Reader

It's refreshing to see an article like this among the many other articles predicting self-driving cars will be here in ~2 years. Volvo has committed to selling level 4 vehicles by the end of this year (in limited locations) - I'd love for them to follow through and see how good their technology really is. I suspect the transition will be far more gradual than most people are willing to admit, and take decades instead of years. Keep in mind, serious research into self-driving cars started in the 80's, and then there was a huge increase in interest in the 2000's because of the DARPA challenges. After that Google took interest, and recently the major automakers have become serious about it as well. But it's worth noting that originally Chris Urmson had predicted Google would be selling fully autonomous cars today, and instead Google lost a lot of their talent in the last year including Urmson and there is no public deadline for selling a product.

I think progress is great and self-driving cars are awesome, but I don't believe they're just around the corner.

View on HN

I took the liberty to draw a few situations which AI would have likely no clue what to do, even if it detected the danger. I'll be adding some more as every day on the freeway brings me new ideas (in fact I'm editing one right now). Something like a "Winograd Schema for self driving cars": http://blog.piekniewski.info/2017/01/19/what-would-an-autono...

Love the artwork!

Nice work! Some of the situations in the drawings imply that the car needs human-level commonsense to perform properly. For certain things like 'fire in a tunnel' or 'tornado crossing the road', there could be sufficient time to warn humans to take control or at least give verbal instructions, while the car is slowing down drastically or even turning away as a safety-first precaution.

The challenge for the autonomous car would be "how to know that it doesn't really know". I wonder if some existing philosophy or theories are applicable for such a purpose in the real world. Does anyone have pointers?

Your artwork is great! I'm not sure #1 would really be a problem as that situation never should occur and is totally preventable (there always should be traffic cones/barricades surrounding the cover, something the car easily could detect). In the rare case it does happen, the utility company would/should be at fault for leaving a manhole open like that.

I can also think of a few ways to prevent number 2 (basically, a combination of GPS + knowing where all intersections occur + road data from thousands of other connected cars = knowledge of where every stop sign is. Certainly not foolproof, but I think ultimately it is a problem that has possible solutions)

The rest of your examples are fantastic though, #3 being quite terrifying actually. They are all very intriguing thought-experiments and I look forward to seeing your future additions!

> In the rare case it does happen, the utility company would/should be at fault for leaving a manhole open like that.

They're who's at fault, but you're the one who's dead.

I'd rather my car avoids holes in the road even if they haven't been marked [yet].

> They're who's at fault, but you're the one who's dead.

This is what my dad taught me too and why I've since always been cautious both as a driver, bicyclist and pedestrian even when I'm in the right to drive/go. Being right isn't as fun when someone's on their way to the hospital/morgue.

An open manhole will not cause you to veer out of control and die in a horrific accident. At worst, you might have a blowout. But in general, unless you have a very small car, most cars tires are larger in diameter than the diameter of the pothole.

Provided that your tires are normal (and not the thin-sidewall don't-my-ride-look-cool tires with big rims), and in relatively normal balance - and you are going a normal speed - it most likely will drop slightly then bounce off the lip, jarring car forcefully, but likely not doing much damage.

If the tire doesn't blow (it may), then you might get a cracked rim (if alloy) or bent rim (if steel and depending on the force and deformation), and you would probably want to have the tire inspected (because the plys may be compromised in the area from the sharp impact, which could cause premature interior delamination of the tire in the future).

Yes - ideally you want you or your car to avoid an open manhole or large pothole, but in general, its unlikely to be the cause of a serious accident.

A sinkhole (named "Steve"?) just opened up in a highway near Oakland, CA. Initial photos showed a hole a couple feet across. Manhole-sized. Repair crews later found that under the asphalt the sinkhole was large enough to swallow a car. Human drivers recognized that something was amiss and avoided the hole. Would a self-driving car?

Let's put "ability to detect solid road in front of car" near the top of the requirements list. Whether it is a pothole, sinkhole, missing manhole cover, or missing bridge, it needs to be a mandatory requirement. From what I've seen of LIDAR imaging it seems like it is possible.

Hey, I just finished another one. A school bus with a swinging open stop sign in the middle of the freeway.

Anyway, just a food for though, autonomy really requires a lot of "intelligence" and our technology is not quite there to deal with all these bizarre corner cases. I'm glad you like it.

I think such things can be discounted by knowing that motorways never have stop signs (same with the prank stop signs). Of course, there are a lot of things that only cause a change in driving behaviour in very specific contexts. The bus' stop sign probably only applies when the bus is stopped. Similarly, in Germany at least a bus may turn on hazard lights on a bus stop, requiring everyone to drive past only at walking speed. A bus stopped on the shoulder of a motorway with hazard lights on is a different context again and thus should not trigger the same behaviour.

There are a lot of rules and laws and they change from country to country, or in the US' case, even from state to state. Self-driving cars must know these things and react accordingly. So I think the scenarios presented here are just a few (admittedly, more far-fetched than others) more contexts amidst the probably hundreds of others that already have to work correctly for switching safely between city and motorway driving, driving in a living street, observing right of way correctly in all circumstances (roundabouts, weird stuff like four-way stops, signs changing ROW for one intersection, or a stretch of road, lowered kerbs, people exiting a living street even though it's to the right, cars on an on-ramp and perhaps letting them in based on how far the on-ramp still continues, ...).

Stop signs are interesting in any case, since they have a characteristic shape. If we go full autonomous, then snow-covered signs must be correctly observed as well, at which point any octagon shape may be a stop sign (perhaps, again, depending on context). Same with signs that don't reflect well anymore at night.

The real sign SDVs are here will be when infrastructure starts accommodating their needs. Humans aren't really good at driving either, so we've invented a lot of ways to help them and direct their attention. Open manholes are supposed to be marked clearly, because people do miss that stuff. When there's snow on the road hiding lane markings, someone will come and clean it out. Signs are made to be retroreflective. Etc.

So at some point, I suppose the infrastructure (broadly understood - including laws) may be modified to reduce the dependence on cultural context and other things machines are weak at. So for instance, it won't be every sorta-octagonal shape that works as stop sign, it will be required by law to be clearly visible and also have some machine-friendly accommodations, and SDVs will be free to ignore signs without those accommodations.

(Doesn't solve the prank problem, but humans are equally vulnerable to a targeted prank anyway.)

> (Doesn't solve the prank problem, but humans are equally vulnerable to a targeted prank anyway.)

I would hesitate using the word "equally". People are actually quite robust. Particularly the second human in row will certainly not be tricked by the same prank that tricked the first one.

I want to emphasize the word "targeted" I used. Pranks involve an intelligent agent with malicious intent and an attacker's advantage - i.e. prankster is free to exploit any vulnerability of its victim. People have different vulnerabities than machines, but they still have them.

Sure they do, agree with that. But word "equally" suggest the susceptibility is the same. I would actually emphasise the difference. It is much easier to fool a machine than a human, particularly if we have a copy of the machine at hand and tinker with it (see adversarial examples for deep nets). Humans are all different, so we can never expect our "adversarial example" to be 100% certain to work.

Especially if it's a familiar road.

> When there's snow on the road hiding lane markings, someone will come and clean it out.

In Nordic countries, you don't see the lane markings for several months, they are under snow and ice. And sometimes, when it melts in April or May, you notice you have been driving on the roadside for a few months :-)

Oh, and you don't see anything at all if you are behind a truck or a bus.

In Nordic countries...

In some parts of some Nordic countries. In my part of my Nordic country we haven't had proper snow for probably about 6 or 7 years and even then it was only for a few weeks.

But more generally. If the snow clearing machines where also driver less then they might be able to run more or them more often and keep the roads clearer.

It's not just a matter of running them often. Where are you going to put all this excess snow? And operating the plows won't be free even without labor.

> Humans aren't really good at driving either,

I'm not sure where did that come from. One fatality per 200 million miles not good? Seriously, there are millions of people travelling every day. And true, there are accidents, but I think this "Humans aren't really good at driving" mantra is not really serious (but frequently repeated recently by PR of some companies).

Attentive humans are extremely good at driving, distracted humans are much worse, but perhaps the technology should focus first on the much easier task of making sure the driver attends. That would probably safe many lives, before we can have a real autonomous car.

>When there's snow on the road hiding lane markings, someone will come and clean it out.

In New England, snow can completely cover the road surface for days or even weeks at a time, and ever-changing piles of snow cover the curbs and parts of the lanes. Humans just choose a path without regard to where the lanes are in the summer. On some roads this turns a four lane road into a two lane road with a lane-width snow pile between the lanes. In a few spots it turns a two lane road into a one lane, with drivers from different directions taking turns.

Remember when Boston was about to start dumping snow into the sea because they ran out of places to put it?

Yeah, there were a few spots in my neighborhood that they gave up on trying to plow, but those of us with 4WD trucks were able to get through. Not sure how a self driving truck is going to know which snowbanks it can drive through/over and which it can't. Sometimes I couldn't tell until I tried. That was fun!

Now I really wish I had taken more photos that year specifically to illustrate this sort of thing.

But if it's that reliant on a special sensor it's quite susceptible to not only pranks but sabotage or plain old maintenance issues.

I know a few places where the stop sign is > 90% covered by bushes in summertime, and the continuous transversal white line > 90% covered by gravel (and the remaining 10% have worn off). Yet people do stop. Either because they know the place, or because they "feel" there is something suspicious (the comparative size of roads, the angle of the crossroad, the fact that there is a "pile" of gravel there, etc.). Not sure how a car could decide this by itself.

I wouldn't say never; sometimes you see them in really odd places.

You could discount that by noticing that stop sign generally shouldn't move relative to the ground. But then, you may want to notice when a police car wants to pull you over by waving that thing they wave out of the car's window...

I actually love trying to make computers deal with the real world - it quickly reveals just how goddamn complicated the real world is, and how many things we think as hard and fast are utterly arbitrary.

Right, there are cases where such a moving stop sign would actually be a real stop sign. The problem is, we try to program in the complexity that cannot be anticipated.

That's something what's easy to train. Each of these are a bit country specific like how a road sign is mounted and where, but this is far from a real problem.

What is the real problem then?

LIDAR imaging already scans the road for it's topography. An open manhole would be easily detected (its equivalent to even a particularly deep pothole - un-navigable).

I don't know how Tesla's video only solution would cope, but they're not Level 5 yet.

How would it show up different than a regular pot hole? What if its almost filled with water and the only thing giving away its true nature is its roundness? The point is to explore the limits. If these aren't the limits, then they are likely very close. I'd like people to think of those cases before they jump into their car and start playing cards (like i one of the videos in this thread).

Basic geometry. A pothole of greater then a certain depth, at a known angle from sensor will produce a different distance profile (namely - rather then an expected arc, you'd see a sudden increase in distance, corresponding to a non-planar surface). Depth is inferred from the deviation from the plane of the road-surface around it.

This is not actually a particularly interesting edge case.

The water-case is more interesting, but I would counter by posing the question of how a human would know whether a water-filled pot hole is safe to navigate? A clear water pothole would not impinge the LIDAR, a murky water one might and could fool LIDAR I suppose (I can not find any solid info on this interface).

Of course car RADAR and a visual camera are likely to also be fitted - both of which could identify a water-filled pothole which LIDAR might struggle with (RADAR can penetrate water and find the unknown backscatter, a camera can simply see the murky-water pothole and choose to navigate around it or stop - in both cases a human can't make a better informed decision).

It depends on the context. Humans may make a better decision if there is something else providing the information. Perhaps the manhole cover sits at the curb triggering immediate attention. Perhaps there are cones, only in the wrong place because some drunk guy moved them for fun. Such is reality.

The problem is not so much availability of information (a sensor packed car has a lot more information than a human behind a windshield), but making sense of that information, particularly in those 0.0001% uncommon, strange cases.

Humans can connect evidence (a manhole cover sitting next to a water filled hole) because they generally know how things work (aka common sense knowledge, long standing problem in AI). Humans can infer what happened and predict what could happen. These things are not really available to AI which is these days more like see-react, not see-anticipate-react. I have a specific post on this on my blog:http://blog.piekniewski.info/2016/11/03/reactive-vs-predicti... The blog has many other posts on limits of todays approach and on some ideas to fix it in the future.

I agree that some of these cases may be challenging to some humans (particularly inattentive), but we want the self driving car to be (much) safer than inattentive driver, so we need to set the bar high.

Clearly if you consider any of these scenarios in isolation we can dream up a solution, but can you do that with literally any adverse scenario? That seems like the point to me.

Yes, that is exactly my point. Solving these particular situations is like typing all the possible Winograd Schema sentences into a chat bot. Certainly possible but does not solve a larger problem.

In reality there will always be a different situation. These situations sit in the statistical long tail (are too infrequent to reliably train, and too frequent to ignore) and vary enormously from case to case. These are just examples.

Except if the first scenario is solvable (fairly trivially) then why should I assume there must exist an unsolvable scenario?

An argument from the vaguery of "humans have context" with regards to driving is a poor one - humans that are driving do not do a good job of analysing context because of reaction times, and do not share a common context they react to similarly.

You haven't made a compelling case, because you've yet to present a compelling example (i.e. one definitely unsolvable by reasonably usable technology on an autonomous platform). The manhole scenario is based on assumption about the operation of LIDAR which simply aren't true and you've had to modify it in a number of ways to try and make it tricky (i.e. when has there ever been a manhole completely full of water?)

I think you're still confused. The point isn't that any of these scenarios don't have solutions. It's that there is a practically infinite number of scenarios like them and it's impossible to plan for each and every one of them.

> In the rare case it does happen, the utility company would/should be at fault for leaving a manhole open like that.

Sometimes, the utility wouldn't be at fault. It has happened in the past the people would steal manhole covers to sell them for scrap (you know, to get money for their next 'fix').

This is less of an issue now, as most scrap companies won't take manhole covers anymore, unless the seller can prove they represent the locality shown on the cover, and that they have such authorization...

...but there are less scrupulous scrap companies.

re: open manholes - Why would an AI have no clue what to do if it could detect the danger? Drive over the middle, drive around it, or stop. I don't think there is any reason to think AI would be worse at handling the situation than people currently are. See this manhole "catapult" video compilation.

https://www.youtube.com/watch?v=ezYV2Aas668

There are lots of places where ambushing drivers to steal them is pretty common. For example, things like throwing rocks from hiding (like a pedestrian overpass) to break some cars' windows and make them stop, and then some accomplices come out and attack. This was really common a few years ago near my home. Now, I think they will just need a stop sign, and do the same as #5.

>Open manhole.

Not sure why this would be a problem? The car can identify a dangerous road surface and navigate around it.

>Many stop signs.

Is it really so bad if the car does stop at every one? It's inconvenient, but not dangerous. Unless there's someone behind in which case the car should know whether there's a safe stopping distance behind and act accordingly.

>Fire in a tunnel.

This doesn't seem like it would be that difficult to detect. And also, I would expect self-driving cars to have an emergency stop button.

>Tornado.

I would imagine this could be detected as a visual anomaly, but more generally... yeah natural disasters suck.

>Potential car-jacking.

I really don't think this should be the car's responsibility. If there's a serious risk of people with guns ambushing you on a road, then it's not safe and you shouldn't be driving there. How the self-driving car reacts to that is the least of my concerns.

> I really don't think this should be the car's responsibility. If there's a serious risk of people with guns ambushing you on a road, then it's not safe and you shouldn't be driving there. How the self-driving car reacts to that is the least of my concerns.

Well, I think the idea behind this prompt was more about how it would handle humans where there shouldn't be any. I would imagine that with the premise that the car would stop for unexpected pedestrians, a car jacking/mugging of this type should be as simple as simply find a remote road, wait for a car with a preferred target, and then intercept - the car would acquiesce and make a person vulnerable to an attack (as simple as smashed windows with a window breaker and then an attack on the occupant.

The picture has an extreme outlier, but criminals adjust to technology pretty fast, especially when it makes their lives easier. It may not be armed gunmen looking to get you, but if you can be assured you'll get a target to stop, I'm not sure why criminals wouldn't exploit it.

edit: change "attach" to "attack" as I meant it originally

If they just want to steal your car, coming to a stop and abandoning the vehicle is probably the safest thing to do anyway.

But if they actually intend to harm you, you've got bigger problems, and you should probably hire a professional defensive driver instead of expecting consumer AI to support your edge case.

I'm not sure I'm explaiing the scenario well - this isn't a planned attack, it's an opportunity attack where you're the victim based simply on the fact that you happened to be there, much like a mugging.

Assume the following for a moment: It is known that, when presented with pedestrians in an unexpected area, an automated car will plot away around them or will yield until they are no longer in the way.

Supposed that with the condition, you were having your car drive you from downtown to your small suburb that requires going down any generally empty road. (i.e., no one is around because it's late) In the distance, 4 people form a loose barrier that the car can't safely pass through so it triggers the logic to yield to pedestrians. The people are muggers, and they quickly break the windows and proceed to mug.

Yes, it's a very specific scenario, but if such a case of logic exists, how long until such a scenario becomes common place? This isn't asking AI to evaluate and protect people from targeted attacks or inventing paranoid delusions of importance, it's about figuring out how to respond to a fairly simple abuse of an often called for bit of logic in the AI.

What do you think a human would do differently in this scenario? If a normal person is driving home late and sees even just one person standing in the road, they'll probably slow down for them before even thinking about it, just like the car. I think if this were such an effective way to mug people, it would already be a major problem.

And again, you're talking about a car that's surrounded by cameras by design. It would be easy to include a button that immediately starts streaming all camera info to cloud storage (or indeed, just do so by default if mugging were such a huge problem).

If we define "normal person" to mean a person who lives in a safe country where carjackings are not commonplace, maybe.

Yes, that is the market I expect self-driving cars to target.

You don't see any demand at all in, say, South Africa, or Brazil? I think that's delusional.

Did I say that? I said I don't think they will be the (initial) target market, and thus the cars inability to deal with carjackers will be a non-issue.

> But if they actually intend to harm you, you've got bigger problems, and you should probably hire a professional defensive driver instead of expecting consumer AI to support your edge case.

I mean this is just reality in a lot of places; I don't see how you can just handwave it away. Cars are designed to operate in all kinds of extremes that probably don't apply to your personal situation.

>Cars are designed to operate in all kinds of extremes

Yes, and self-driving cars will not be suitable for those extremes. I don't see how that's a major problem?

I don't see a lot of demand for a car that's only appropriate for pleasant Sunday drives.

There's a huge range of common usage between "pleasant Sunday drives" and "extreme defensive driving against armed goons".

I don't think the latter is necessary for self-driving cars to be successful.

But it's just an example, is the point. We don't have big carjacking rings, but we have plenty of extreme weather, old roads (even dirt roads depending on where you are), and other extreme circumstances that maybe don't matter to somebody who never wants to leave a forty-mile radius of Mountain View.

Even in an idyll you could imagine some novel circumstance created by, say, downed power lines.

This is a method of committing a carjacking that works today, it's not some kind of new exploit. Very few people will intentionally run down a person in the road, as opposed to stopping, because they think something fishy might be going on.

The scenario I'm familiar with involved a bicyclist crashing, or laying down, his bike on a slow stretch of road in an industrial area. His companions would approach the car when it stopped. The deterrent for this is a harsh criminal penalty, not AI.

It never became a commonplace crime, in spite of being relatively simple.

I think you can get on Liveleak and find lots of video evidence that there are places where drivers will not stop for precisely this reason.

Rather than train automotive AI to handle this case, we should just stipulate that if you live in a godforsaken place that might require you to run over somebody during your drive in order to survive banditry, you should keep your seatbelt fully fastened and remain aware enough to retake control of the vehicle at short notice should it stop. I'm sure some airline has a sign that could be reused for this purpose.

Or, you know, you could just drive yourself. Or pay a driver. Or stop fantasizing about embarrassingly absurd stuff that has nothing to do with the efficacy of automotive autonomy.

The entire point of the article you're discussing is that there are a million bizarre little circumstances like this that the software will probably fail to take into account, not that carjacking specifically is unsurmountable. This stuff isn't "embarrassingly absurd;" it happens daily.

> The entire point of the article you're discussing is that there are a million bizarre little circumstances like this

In defense of the article, which is pretty reasonable, it doesn't mention the ridiculous hijacking example you were harping on. That is quite a unique situation, technically and ethically.

The other examples you gave: bad roads, downed power lines, weather, and fire are all much more reasonable examples, with much more straightforward solutions available. It's essentially obstacle avoidance and exception handling. The article's example of situations involving not having any safe place to stop is even more interesting.

edit: I was referring to TFA, not to the artist who illustrated some stuff on his blog and shared it here. Which was also a fine effort...

> In defense of the article, which is pretty reasonable, it doesn't mention the ridiculous hijacking example you were harping on. That is quite a unique situation, technically and ethically.

It is clearly one of the implied reasons a bunch of armed men would be standing around on the road in that picture. What makes it "ridiculous," exactly?

> What makes it "ridiculous," exactly?

If you're on a road with armed men with hostile intent, the fact that your autonomous car is unable to offer a solution is absolutely the least of your worries. The unique properties of the armed men on the side of the road problem are not representative of the more general problems vehicle autonomy involves. Take your pick.

Speeding past or turning around are sensible actions a human driver could take that the AI probably would not. The whole point of that example is that the appropriate response to armed guys on the road is not the same in one context as another.

> This doesn't seem like it would be that difficult to detect. And also, I would expect self-driving cars to have an emergency stop button.

That it if the passenger pays enough attention, which they won't. Also note: each one of these cases can be programmed indeed. But that is like fighting Wingrad schema by typing in all possible sentences. In reality none of this will happen, but something else will, which we don't even anticipate.

Assuming you've accounted for these scenarios, yea, everything may be just fine.

I think the point is that there are millions of different things that can go wrong, and many have a 1 in 1000000 occurrence. Human intelligence is able to improvise, but what will cars do?

> Human intelligence is able to improvise, but what will cars do?

Halt, and send a warning signal to any vehicles nearby. Which is better than what we can ordinarily accomplish today in those one in a million occurances.

Great artwork! Coming from a country with somewhat moderate weather, I have to ask though:

> Tornado is crossing the road.

Do people really drive when there are tornados nearby? Shouldn't they be hiding in a cellar, or something?

Tornado watches tend to cover large areas of potential hazard and not everyone within an area will receive advanced warning.

I've driven by two (small) tornados while travelling long distances over the last 15 or so years.

I live in Arizona and I drive through twisters like that all the time. They are a constant feature in the desert. Along with tumbleweeds blowing along, which disintegrate nicely when hit by your car, but would appear as a large solid mass to the autonomous system and probably freak it out (as it should).

I think the missing piece here is commonsense reasoning (I really like your art work by the way!). There has been a lot of work in commonsense reasoning in symbolic AI but not sure if anyone is working on that in machine learning.

Yup, there is a boatload of low level knowledge missing from any AI that we build. It is painfully visible with robots (and autonomous cars are robots as well), see DARPA challenge video which was then lead BTW by Gill Pratt: https://www.youtube.com/watch?v=g0TaYhjpOfo&ab_channel=IEEES...

A lot of stuff we take for granted such a supporting oneself by a nearby wall when loosing balance, is completely out of reach of todays "AI".

Really cool page. I think the journalists are missing that the remaining 5% of situations to be solved are much harder than the 95% of common place situations.

That's awesome. Thought-provoking for sure.

Regarding the "driving into a manhole" scenario:

This is unlikely to cause a problem with a car's direction or motion of travel; at worst case, it may cause a blowout, but tires and rims are surprisingly tough (purposefully). I've hit potholes as large as manholes, and other than being very surprising, no damage was done.

Then again, I drive a pickup truck - I wouldn't expect something much smaller to handle as well.

Still, the rate of speed and balance of the car would all play into the scenario. While it would be better for a car to avoid an open manhole (or pothole for that matter), it generally isn't a crazy scenario if the car hits one, either.

If you want to see and hear about crazy stories of mishaps people have, yet the car continues to be mostly drivable, check out the sub-reddit "Just Rolled Into The Shop":

https://www.reddit.com/r/Justrolledintotheshop/

You'll learn both just how stupid people are with their cars and driving, as well as just how robust vehicles actually are.

These are interesting, and clever, and I agree difficult, but I don't think they necessarily need to slow down the progress of self-driving cars much at all.

Tesla's data so far suggests that their current autopilot implementation is reducing crashes by 40% - http://www.theverge.com/2017/1/19/14326258/teslas-crash-rate... - and while these cases are all problematic, they're all fairly rare, and the cases in which they come up and the car reacts wrongly and that's a major problem are going to be even rarer. On top of that, self-driving car performance is only going to improve.

It doesn't really matter (in terms of the value of self-driving cars - there's a mostly independent marketing question) if there's a couple of cases where self-driving cars make the wrong choice and kill you, if there are many thousands of cases where they save your life. It's effectively changed a great many risky situations into some new and different but less likely ones.

If driving a self-driving car significantly increases my odds of survival while driving, in addition to giving me huge amounts of bonus free time, then I'm definitely interested, regardless of risky but rare cases like this.

My great worry with self-driving cars is that they shouldn't be right around the corner, but they might be anyways. Marketing departments might move faster than engineering, smooth-talking the buyers into taking uninformed risks in order to expand the company's training data.

If NHSTA does its job, this shouldn't be an issue in the USA at least.

I know they can deal with the risks, but what about the "informed" part? My pet peeve with Tesla's marketing dept is that there are tons of Tesla drivers who seem to think the autopilot is more capable than it is: https://youtu.be/qnZHRupjl5E Those people are surely not aware of the risks they are taking, so I'd bet even the more responsible users aren't aware of its limits.

OMG that video. I thought people learned their lesson after the FL man's death. The culture is even worse now!

Cars led to more deaths than people had on horseback. I'm starting to feel autonomous vehicles will be the same sort of innovation. More elderly/younger people could use them, and ultimately more deaths will result as people's carelessness increases.

Tesla is still cashing in while resisting safety features like requiring hands on the wheel. Meanwhile, accidents like the death in Hong Kong go mostly unnoticed in the US.

Wow morons, I don't mind them killing themselves, but I do mind them killing someone else.

Wow, that is a scary video.

I'm surprised the NHTSA doesn't have a rule saying auto-pilot should be 100% autonomous, in control, and capable of driving in the current conditions, or it should pull over, turn itself off, and put the car in manual control mode. Counting on drivers to understand when auto-pilot can and can't handle driving conditions seems like a completely stupid idea and a recipe for disaster.

Holy crap. In some sense it is really surprising there's only been one fatality. Just as well tomorrow there might be 10 more if people really do such stuff...

There have been more than just the one fatality. I know of at least one other, but who knows if that's the extent. And there have probably been plenty of crashes and injuries.

> There have been more than just the one fatality. I know of at least one other, but who knows if that's the extent. And there have probably been plenty of crashes and injuries.

Yup. The Tesla accident in Hong Kong (or Heibei province?) [1] seems like another likely case. We only know about that because the driver had a dashcam that saved the video, and the parents found a lawyer to sue. Tesla claims they don't have data on whether or not autopilot was active at the time of the crash. Yet the dashcam survived..

Tesla and others need some regulation with teeth. They should be required to have black-box style data recorders. They should also be required to report accidents involving autopilot.

Currently, car companies are not required to report accident rates involving autonomous-driving modes in most states. I think only California and a couple others require it.

[1] https://electrek.co/2016/09/14/another-fatal-tesla-autopilot...

Tesla:

> Because of the damage caused by the collision, the car was physically incapable of transmitting log data to our servers and we therefore have no way of knowing whether or not Autopilot was engaged at the time of the crash.

Well that is even more troubling. Looking past the suspicious lack of logs from the car in the incident, it should be a rock hard lose-your-business-license requirement for any kind of autonomy in cars, that the log data is preserved in accidents. I can imagine very few scenarios where a black box type of device could not be salvaged from a wrecked car.

EDIT: From the wording, it seems like it may actually have been the 'transmitting' of the data that was impossible. That would make things even worse, if Tesla could not even be bothered to requisition the car data (through legal means if necessary) in a case where a bug in their software may have killed someone.

Its interesting, I vaguely remember reading a report I think by the NHSTA that discusses how, despite the ability for accidents by users not using the features properly, these features still result in a net gain in safety

The Trump administration stated today that they will remove 75% of all regulations currently in place.

Given that self driving cars are a potential job creator you might find those regulations removed.

> Given that self driving cars are a potential job creator

This is the first time I’ve seen someone refer to automated cars as a job creator. It’s almost always presented as the thing that will kill the trucking and taxi jobs by the millions.

In the next 5-10 years (maybe longer ?) it will be a net job creator.

And then a massive job destroyer. Of truly scary proportions.

I fear that myself. Stake holders in the autonomous car business should lobby against that because in the end it's in they're own best interests.

I literally just came to HN after browsing slate.com for a bit, currently on their front page is an article about a railroad company that sent 86% of their lobbying money to Trump and GOP PACs this last election cycle. Half of this company's revenue is based in pesos and susceptible to dollar/peso fluctuations and its core business is related to moving goods back and forth between the US and Mexico. Trump took dead aim at free trade, and NAFTA in particular on the campaign trail. Self interest seems to play no part in who people (and companies, thank you Citizens United) choose to support.

http://www.slate.com/blogs/moneybox/2017/01/23/kansas_city_s...

Edit: expanded on why a for profit company being pro-Republican would be against its own self interest.

Hmmm, so as cars get cheaper (no driver needed) maybe we'll see mobile billboards by the thousands.

I wonder how many other possible unintended consequences there will be

What a horrible waste of work that would be. In the sense that the energy and time taken from other road uses just for a little advertising would be obnoxious.

Adverts only continue to exist because they ultimately find the sweetspot between inconvenience and payoff. A road mobile billboard is inconveniencing everyone and the payoff is nil. In contrast to a YouTube advert where the payoff is the youtube video, a mobile billboard has no payoff.

So like the trucks going up and down the Las Vegas strip, but hundreds and hundreds of them . Moving billboards everywhere. Unintended consequences indeed.

Level 5 is a long way off, but level 4 really is right around the corner. The article was talking about level 5.

> level 4 really is right around the corner

I disagree. Level 4 does not require driver attention. That is still a long way off.

From Wikipedia [1]:

Level 4: The automated system can control the vehicle in all but a few environments such as severe weather. The driver must enable the automated system only when it is safe to do so. When enabled, driver attention is not required.

Level 5: Other than setting the destination and starting the system, no human intervention is required. The automatic system can drive to any location where it is legal to drive and make its own decision.

[1] https://en.wikipedia.org/wiki/Autonomous_car

Yes, so far at least, I don't think any company is even proposing anything past Level 3 on their near-future roadmap.

By my reading of the article, stale2002 is correct. Pratt is clearly talking about the difference between Level 4 and Level 5 in his first two answers. Level 4 doesn't require driver attention, it just requires the driver to be (instantly?) available if the system has a problem, and this change-over is the situation that Pratt worries about in his fourth answer.

(NB: When he says "The article was talking about level 5", it's clear from context that stale2002 means that the article was talking about level 5 in terms of technological difficulty and timelines.)

> Level 4 doesn't require driver attention, it just requires the driver to be (instantly?) available if the system has a problem

Sincerely asking. Doesn’t that mean it requires driver attention?

I don't think so, mostly because I don't think a qualifying system would actually require an instant switch over. I can't find anywhere that it the timescale is defined; if I had to guess, I'd say it's defined in terms of whatever the minimum safe timescale is (empirically), i.e., Level 4 is whatever it needs to be to allow a human to read a book safely, but not necessarily sleep.

For instance, a Level 4 car probably can't deal with unexpected road construction, and might simply bring the car to a halt, smoothly but quickly. It would be dangerous if the driver was asleep and the car stopped dead on the highway for the 30 seconds it takes someone to wakeup and get oriented, but it could be pretty safe if the driver just needs to look up from their movie and grab the wheel.

That's just speculation, though.

Level 4 doesn't require driver change over. Level 4 is "it works perfectly on highways and in major cities". Level 4 is awesome!

Level 5 is "all cars in the entire world don't have steering wheels anymore". Yeah, level 5 is hard, but who cares, level 4 is good enough.

So basically by your definition, level 5 will never ever happen? Seems a bit useless then TBH.

Also, as pointed out in another comment, Level 4 does require driver changeover, it's just that the car is supposed to safely handle the event of driver not responding to that request (e.g. by stopping in the middle of the road).

> So basically by your definition, level 5 will never ever happen? Seems a bit useless then TBH.

Well, even trains on tracks have humans ready to pull the break if something is wrong up ahead.

Would you ride a train that had an accident caused by something for which its driverless system was not prepared, but that a human conductor could have avoided?

I think it's possible level 5 will happen. There will still be accidents on roads where only driverless vehicles exist. Some people will choose to ride in such vehicles, and some won't.

No, what I'm trying to get at is GP post changed the definition of level 5 from "some cars don't have steering wheels" to "all cars don't have steering wheels". A non-self-driving car will (by definition) always be cheaper than a self-driving car, and some people just like driving or distrust machines. Together with the fact that (especially in the United States of Freedom) governments will never ban cars with steering wheels, that definition of level 5 will never happen.

Oh, yeah, I totally agree. For cost reasons alone, the world will never be uniformly automated.

That's weird to change the definition that way. The levels describe an individual car, not all cars.

> A non-self-driving car will (by definition) always be cheaper than a self-driving car

I don't think that's true at all. A self-driving car can save money on not having to have a steering wheel, pedals, hand-brake etc. Possibly even won't require airbags or seat belts if safety is improved massively

non-self driving cars could become expensive luxury cars

Surely you jest; that the cost of the wide array of self driving sensors, the data input subscription and the hardware and software to run the self-driving system is cheaper than a few mass-produced bits of plastic and metal?

Not right now.

Maybe in the future. Especially 'hardware and software to run the self-driving system' will cost a few dollars (small computer, zero amortized cost of software)

> Pratt is clearly talking about the difference between Level 4 and Level 5 in his first two answers

No, Pratt is saying that manufacturers are hyping level 4 as being around the corner, but it is not. Pratt does not think the claims for level 4 are sufficiently backed up:

"...That’s Level 4. And I wouldn’t even stop there: I would ask, “Is that at all times of the day, is it in all weather, is it in all traffic?”

> Level 4 doesn't require driver attention, it just requires the driver to be (instantly?) available if the system has a problem

How exactly would that work? You don't have to be attentive but you must be available to take over?

At any rate, this is not the definition provided by the Society of Automotive Engineers [1], which is the definition NHTSA has adopted for levels of autonomous vehicles.

SAE says for level 4:

"the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene"

In other words, the system could request human intervention, but it will still do something even if the human does not respond.

Wikipedia appears to have either summarized this in different words or has a different version.

> this change-over is the situation that Pratt worries about in his fourth answer.

I wholeheartedly agree with Pratt that "In some ways, the worst case is a car that will need driver intervention once every 200,000 miles..."

[1] https://www.sae.org/misc/pdfs/automated_driving.pdf

> "...That’s Level 4. And I wouldn’t even stop there: I would ask, “Is that at all times of the day, is it in all weather, is it in all traffic?”

But that doesn't actually contradict stale2002 or diminish the enormous potential economic importance by more than (say) a factor of 2. Level 4 systems that are available to use 60% of the time are still revolutionary, and compatible with both stale2002's and Pratt's comments.

> How exactly would that work?

See my comment to function_seven for my speculation. Although I used the Wikipedia definition, I believe everything I said is compatible with the definition from SAE you have quoted. In particular "the system could request human intervention, but it will still do something even if the human does not respond" does not conflict with "it just requires the driver to be available if the system has a problem" because the "something" the system may do is bring the car to a rapid (or even emergency) stop. This can become dangerous if the human can't resume control within a few seconds, but is an easily acceptable risk to occur once every 100k miles as a price of Level 4 autonomy.

> But that doesn't actually contradict stale2002

stale2002 said the article is talking about level 5, and that level 4 is around the corner.

Yet the article is talking about level 5, 4, and below. It is saying some car companies are overhyping their capability to reach levels 4 and 5.

Pratt feels even level 4 is not right around the corner.

> Level 4 systems that are available to use 60% of the time are still revolutionary

To reach a certain level, you must operate under the definition of that level 100% of the time. You can't be level 4 60% of the time. That's level 3.

I agree it would be revolutionary to be at level 3. Right now no car is there. They're all at level 2, which require human monitoring of the driving situation.

> To reach a certain level, you must operate under the definition of that level 100% of the time. You can't be level 4 60% of the time. That's level 3

No. I'm looking at the SAE definitions, where it says

> A particular vehicle may have multiple driving automation features such that it could operate at different levels depending upon the feature(s) that are engaged.

clearly indicating that the levels describe modes of operation, not immutable car classifications.

Regardless, this has just become a semantical dispute. stale2002's comment makes the most sense under the interpretation "cars that that can drive at Level 4 60% of the time are just around the corner", and that comment is a valuable counterpoint to the reasonable interpretation of Pratt's interview.

> the levels describe modes of operation, not immutable car classifications

Look at the column where human attention is required ("Monitoring of Driving Environment"). Every car in existence requires human monitoring in every driving mode.

The moment attention is not required, the car company becomes liable. Volvo will release a level 4 car this year where they assume liability, however, that is only for a hundred people in Sweden.

"Around the corner" means widely available, and that's just not the case.

Right now Tesla does not reliably save the data from its accidents. How could a car company assume liability when they can't even save the data?

Yeah, exactly, 60% level 4 is still a crazy awesome future that I want to live in.

> Level 4 doesn't require driver attention, it just requires the driver to be (instantly?) available if the system has a problem

Based on my best understanding of the article and the good description on these levels in the CES keynote linked in the article...

This is Level 2, when the driver must be available instantly. Level 3 is when there driver must be available but there's plenty of time (the article mentioned 15 seconds) to swap.

Level 4 should not need any human intervention when engaged. It can only be engaged when it's safe to do (e.g. in a city in good weather conditions) but once it's enabled, it fulfills the task given to it without ever needing to fall back to human hands. You should be able to sit on the back seat drinking beer. Level 4 should be able to safely stop the car and wait for assistance if things go bad without driver intervention.

Level 2 is where the cars out there are now. Level 5 is a long way away.

Google can almost drive a half-year to a year's worth of driving for the average individual without any intervention at all. That is very impressive, and their streak is growing fairly quickly. Of course, that would probably be done in less than a week as a ride-share service, but consider that any intervention may also be predictable beforehand (e.g. weather conditions).

The article was talking about Level 5 as a long way off but it also spent time talking about a lack of consistency in how these levels are applied [1]. Some systems might state they are Level 4 but that's only true for well mapped good weather locations so one might legitimately wonder if such a system is truly Level 4.

The article is mainly about: Level 5 as difficult, inconsistent application of Levels, the fact that rare difficult situations carry most of the valuable information for a learning system, preventing mismatched expectations, untenability of systems which require human attention and vigilance. The final point made was, instead of focusing on self-driving cars, focus on cars which complement human ability. For example, humans are good at theory of mind and inference in rich world models while machines are better at total situational awareness.

I'll add another. Self-Driving cars need models which at the very least, are able to capture uncertainty over predictions. This makes them more robust to glitching catastrophically in parts of the state space that are novel to them.

[1] > Yes, so NHTSA had 1 to 4, and SAE was very smart, they said we’ll take Level 4 and we’ll split it into two things, 4 and 5, and the difference is that Level 5 is everywhere at any time, and 4 is only some places at some times. But otherwise they’re basically the same. The SAE levels are fine, but people keep making mistakes. And so Level 2 systems are often called Level 3, which is wrong. A key thing of Level 3 is that the person does not need to supervise the autonomy. So no guarding of the machine by the human being.

> level 4 really is right around the corner

This is a little misleading. Level 4 is full autonomy within certain domains. So actually Level 4 is already here today -- on test campuses. The meaningful question is when is Level 4 coming to cities. Maybe not decades but definitely not "really right around the corner".

>but level 4 really is right around the corner

I work in the industry. This is not true. We haven't even completely solved Level 3 yet.

I always thought that the first real application for level 4 would be highways. Aren't we really close to that? They should be much easier than urban driving.

Reply on news.ycombinator.com