I Just Drove Eight Hours on Tesla Autopilot | Hacker News Reader

I Just Drove Eight Hours on Tesla Autopilot

bloomberg.com

68 pointsnreece75 comments

Put aside the silly title and the inconsistency between the subtitle and content ("the only real problem was the driver behind the wheel" vs "The sensors failed to register a rapidly accelerating gold-colored SUV beside me and would have driven directly into its path had I not quickly taken over") and this article has a really good point to make:

As we get closer but not entirely to autonomous vehicles, figuring out how to communicate to users (at what point do we stop calling them drivers?) the necessary level of engagement (and notifying them when that level changes) is going to be increasingly critical.

> the necessary level of engagement (and notifying them when that level changes) is going to be increasingly critical.

Is there an "uncanny valley" style term for this?

I've been calling this the "deadly valley". The automation is good enough that the driver will tune out, but not good enough for it to be safe that way.

There should be, and the best candidate term is "Children of the Magenta", a term describing pilots who are too dependent on the Magenta lines on their instrumentation.

99% Invisible did a good podcast episode on this a year ago: http://99percentinvisible.org/episode/children-of-the-magent...

Their follow-up podcast episode talks explicitly about this problem (first seriously discussed in the context of airplane automation) becoming important in the context of self-driving cars, and how Google is very aware of this (hence Google's push for 100% automated cars, since the middle valley is so dangerous).

There are a couple of ways you can do this on the water (radar, AIS), but the names for doing so escapes me. In ocean navigation there is the notion of a tripwire: if X happens, do Y to get your shit together. Ocean traffic also sometimes has the issue of many-thousand-ton vehicles on autopilot while the crew are elsewhere or asleep (not all the time, but enough to be unsettling). Then things get fantastically complicated as you near the destination.

There's no inconsistency between those two statements. And "figuring out how to communicate to users the necessary level of engagement" doesn't solve very much either. For example, if the car's sensors can't detect a rapidly accelerating gold-colored SUV, how will the car ever be able to tell you "ok, now it's time to watch out for gold-colored SUVs"?

Ultimately it comes down to what a different commenter pointed out, that drivers need to have adequate understanding and expectations around this car and its capabilities.

The term "driver" will still fit. We apply it to someone guiding a camel to a destination.

I think the point is we're trying to make cars into high-energy horses or camels. I'm thinking of the ponies in Iceland that get their farmers back home after a night heavy imbibing. So long as you got on the right horse, it'd take you home. Yah Tesla!

I find myself thinking about the opening scene of a spaghetti western, where a guy is sleeping on a frame being dragged across an arid landscape by a horse.

A car that only requires the effort displayed in "They Call Me Trinity" would be fine by me.

>>The term "driver" will still fit. We apply it to someone guiding a camel to a destination.

No we don't. People ride camels, not drive them. In fact, that's a perfect illustration of the point the parent is making. The reason one doesn't drive a camel is because a camel has a mind of its own. You guide the camel to a specific destination, just like you guide a semi-autonomous car to its destination. In both situations, you may need to take direct control, either by yanking on the reins or grabbing the steering wheel, but for the most part, the vehicle/animal does its thing.

You may be thinking, "wait a minute... we ride bikes too! Does that mean they have a mind of their own?" I would posit that bikes and motorcycles are are exceptions - the reason we ride them is because doing so resembles riding a four-legged animal.

I think OP meant the person who stands by the camel guiding it (for example, if the camel is being used to carry a load, rather than a human).

For example, see this image: http://imgc.allpostersimages.com/images/P-473-488-90/40/4000...

Or this image: http://imgc.allpostersimages.com/images/P-473-488-90/64/6424...

EDIT: I like how your username fits your comment so well. :-)

In some ways, it could be useful to have people still driving even when a car is fully autonomous. The human input could function like a copilot, and the autonomous system might be able to learn from the actions of human drivers. It also helps people feel like they're in control.

>>> As we get closer but not entirely to autonomous vehicles, figuring out how to communicate to users (at what point do we stop calling them drivers?) the necessary level of engagement (and notifying them when that level changes) is going to be increasingly critical.

For starters we could not call advanced lane assist software "autopilot" and we also should require hands on the wheel sensors until the automakers assume liability.

I agree that Autopilot was the wrong choice of name, I'd say it's reckless bordering on seriously negligent.

The name was chosen on its marketing merits not based on its relation to the actual function it performs. That might be ok if you're designing a feature for the car stereo, not so for "Cruise Control v2.0"

"Consider this: If a Toyota driver had standard cruise control set for 70 miles per hour on the highway and failed to take over and reduce speed for a 25 mph turn, would we blame the cruise control for the resulting crash? Relinquishing full control to Autopilot is no different."

That's why Toyota calls is cruise control and not autopilot

But cruise control only means cruise control because someone decided to call that feature cruise control. I find it harder to blame Tesla for an ambitious phrase rather than drivers that are considering nothing beyond the name of the feature before turning it on.

I'd say the bar for writing it off as simply an ambitious phrase is a bit different when you know that particular form of "user error" is literally lethal.

So far one person has faced lethal consequences. As the article states, that person was fully aware of how autopilot worked. Do you honestly think that if that feature was called something else that person would still be alive?

> So far one person has faced lethal consequences.

This is the tip of the iceberg.

How many other Tesla owners aren't fully aware of how the feature works and are just relying on the vernacular usage? Even if 100% of Tesla owners are 100% aware of the feature's capability, what happens when a friend/coworker/spouse/child of the owner borrows the car and uses the feature based on their misunderstanding of the technology?

>An autopilot is a system used to control the trajectory of a vehicle without constant 'hands-on' control by a human operator being required. Autopilots do not replace a human operator, but assist them in controlling the vehicle, allowing them to focus on broader aspects of operation, such as monitoring the trajectory, weather and systems

Autopilot doesn't mean fully autonomous. It was never that originally in aviation either, basicLly just controlling altitude and speed. I'd say Tesla's use of "autopilot" is pretty spot on.

The difference is that "keep height, speed and direction" is a pretty safe way to operate an airplane at altitude: there are no sudden obstacles, and there is a lot of infrastructure to make sure traffic stays nicely separated. A human reaction time of a few seconds is fine.

It's not for a ground vehicle in normal environments.

Agreed, though Tesla's autopilot feature explicitly makes it clear it's not self driving. It's still an assistance, albeit a very good one, somewhat true to the definition that an autopilot system "doesn't replace the human operator".

I do think Tesla have their work cut out in distinguishing what autopilot actually is vs the general Persephone that autopilot means "it controls itself, autonomously".

I agree there's a lot of (potentially dangerous) ambiguity in what the name "autopilot" means and implies.

Personally, I'm not a fan of the name "autopilot" for 2 reasons.

Firstly, the name is sexy and filled with hype. Even for a fully autonomous "driver go to sleep" level 4 driving system, "autopilot" would be a great sounding name (much better than "full autonomy" or anything more explicit in my opinion).

Secondly, the name deviates from industry semi-standard naming such as "cruise control" or "lane assist". I don't believe Tesla's automated features are more advanced than the rest of the industry[0], but the name sounds more advanced. Maybe this is good marketing, maybe this is dangerous and deceitful, maybe both.

[0] I'm not an expert on the state of self-driving systems. If anyone here who is reads this and has some evidence that Tesla's autopilot features are more advanced that what's offered by Audi (for example), I'd love to hear why / see some sources.

If I'm not wrong, autopilot can follow set route also and if the airport supports it, it can land autonomously too. Tesla's use of the term is highly misleading.

No, but granted we're getting into semantics. Feature 1 you describe is provided by the Flight Management System[0], and feature 2 is provided through Autoland[1]. [0]: https://en.wikipedia.org/wiki/Flight_management_system [1]: https://en.wikipedia.org/wiki/Autoland

Get a focus group of 100 average Joe consumers.

Ask them whet they think and autopilot feature is.

The vast majority of them will say. "Err... a car that drives itself"

Of course I'm speculating. But if someone did run such a focus group and the majority didn't give a definition which boiled down to "a car that drives itself" then i'll eat my hat.

The pedantic "true meaning" of a term doesn't matter. Particularly when naming a feature that not only has the capacity to kill, but has now actually done so.

when you call it Autopilot and not SuperiorCruiseControl you've set expectations. It's no use trying to re-define the term more accurately after the fact. They'll survive but the halo got dented.

The Professionals® that actually use a real autopilot tell us that Tesla's is aptly named-- it's limited in its functionality and is designed to assist the operator of the machinery and to reduce fatigue.

The Problem® is one of public perception/branding. Hollywood has convinced the overwhelming preponderance of the population that airplanes on autopilot can taxi away from the terminal, navigate to the runway, avoid collisions with buildings, vehicles and other planes, line up on the runway, take off, navigate to the destination, line up with the correct runway from the correct direction, land the plane, and taxi to the final terminal.

Naming your not-fully-autonomous car driving software "autopilot" is going to have a perception problem. Educating the public after the fact is a hard battle.

Auto-pilot => autonomous piloting.

http://www.merriam-webster.com/dictionary/autonomous

3. Existing or capable of existing independently. But no its totes a public perception problem.

Are you implying that the aviation industry has been using this term incorrectly for a century and counting, and that we should believe a layman's dictionary over actual common usage?

Yes. Cars aren't sold to people in the aviation industry, they're sold to laymen.

Since actual real-life autopilots nearly universally don't act that way, does that not mean it's a perception problem?

If people are dying, then it's a people dying problem. If the number of people who die depends on what name Tesla chose for the feature, then Tesla is at least in part responsible for the people dying problem.

If I'm a bartender, and I tell a customer "this is a strong pour - it's literally cyanide!", and the customer drinks it, there is not a "he should have known the right definition of literally" defense.

So far it seems to be safer, not more dangerous, than regular driving.

I would argue that we do not have conclusive data. If Joshua Brown had his wife and kids with him in the car the simple math everyone is doing (fatalities / miles driven) would not look so good for autopilot. It's going to take a lot of miles before we know for sure. Probably 100x what they have so far.

In an abstract academic sense, maybe. The fact is, that's what the vast majority of people think autopilots do. You can either spend billions on ad campaigns and such and still probably only reach 10% of the market over the course of a few years, or you can just call the feature something else.

In this case, the popular usage of a term used to describe a feature to the general public is the most pertinent meaning of that term.

"We're using industry jargon rather than the popular meaning" just doesn't cut it when your audience is the populace, not industry. This is not Hollywood's fault, it's Tesla's.

Read the post-mortem of the Air France flight from Brazil to Paris. The pilots did not even know the basic error recovery conditions of flying the plane they were in (recognizing dual input, recognizing a stall) because they so rarely flew the plane. Auto pilot can handle navigation to destination, lining up with the runway after being given the approach vectors by the pilot that he/she received from ATC.

If auto pilot in Tesla were used like auto pilots in modern commercial air travel, it would cover 95% of the operation time of the cars.

One pilot, singular, did not know how to recover. The other two did. The one who was at the controls attempted to perform a correct recovery. He was thwarted by the unbelievably idiotic design of the Airbus control system, in which conflicting inputs are simply averaged, and the only indication that something strange is happening is a warning light.

Edit: regarding the pilot who did the wrong thing, I don't think it's fair to say that he didn't know how to recover, either. He almost certainly knew how to recognize and recover from a stall. He was a glider pilot, where stalls are a matter of course. It seems that he panicked, which prevented him from thinking clearly and taking the proper actions. People do that sometimes, and a big reason you have two pilots is so the other one can step in when one of them gets stupid. The fatal flaw with the Airbus was the system which failed to adequately inform the other pilot of the first pilot's stupid actions. (In any sensibly designed aircraft, the two sets of controls will move together, providing physical feedback of each pilot's inputs to the other pilot.)

>and the only indication that something strange is happening is a warning light.

In addition to the warning light there is an audio warning "DUAL INPUT" and tactile feedback (stick vibration). The fact that the pilots did not recognize this condition feeds into my point about them not flying the plane nearly often enough to actually understand how it behaves in anything other than an ideal condition.

It looks bad that the Tesla didn't detect the SUV. But on the flip side, if the SUV had also had Autopilot, maybe it would have been a less dangerous situation. And when more cars have autopilot systems, maybe they'll be better able to communicate their proximity to one another.

The problem isn't autopilot, it's overconfidence in autopilot. The driver who was killed was allegedly watching a movie and speeding when he crashed. Tesla's response is that autopilot is still in beta and people shouldn't rely on it too heavily, oh and by the way here's a story of a man who's Tesla drove him to the hospital after he had abdominal pain. There are plenty of Youtube videos of people screwing around while using autopilot, with a few near misses when autopilot screwed up.

When building a several ton vehicle you need to design for human flaws, and Tesla hasn't. Every other manufacturer with adaptive steering and autopilot-esque features has created attention checkers to ensure the driver doesn't rely too heavily on autopilot. If one of them released a beta autopilot the NHTSA would be on the phone in record time telling them to knock off the foolishness.

Tesla makes very cool cars, and the technologies they and other autonomous car companies are developing will absolutely save lives, but it's stupid to ignore the human factor until you've developed a car that 100% doesn't need a human driver.

The author states that autopilot made him a better driver but I can't see how.

If the car is doing most of the work that requires a person's attention then they will pay attention to something else.

This will prove fatal when the sensors fail or there is some unforeseen condition.

Hunans are, at this point, able to detect and possibly correct for for more errors. Sure the car would be able to react more quickly but maybe it won't react at all.

Am I wrong in thinking that the problems complained about here are precisely because it uses sensors and a camera and not lidar?

(I did a bit of research, and reading around says "probably", but i trust HN's opinion a bit more :P)

Lidar alone is not enough, as it relies on the certain reflective properties of objects around (and there's also solar light noise).

But yes, they ought to add a lidar and more cameras so that there's 360 coverage with a good overlap, and a single failed sensor does not disable the ability to auto-drive safely.

Thinking a further bit down-range there probably ought to be Smart Roads that cooperate with the vehicles to a) keep them apart, and b) keep the moving in the proper way.

They exists, they are called tracks and have trains running on them.

Joking aside, there probably will not be special roads since the cars need to be safe in case the roads fail anyway. And I don't think anyone will be for these new roads that no cars can use so it is also a chicken and egg problem.

The situational awareness of this thing is not good. The author points out that it didn't notice a rapidly approaching car prior to a lane change. The only side-looking sensors Tesla has are ultrasonics, which are too short range.

The Tesla sensor suite is:

- a bumper-level 2D radar, good for not rear-ending cars in front. Not good for obstacles at windshield height. Will detect a car. Might detect a bicycle. Won't detect most pedestrians.

- a forward facing camera using MobileEye software. This 1) recognizes lane lines, 2) tries to range the car ahead by putting a rectangle around the recognized rear end, and 3) reads speed limit signs. It may be able to recognize pedestrians. (Unclear. See [1])

- some ultrasonic sensors which detect obstacles to the side and rear at short range. Mostly useful for parking.

This just isn't enough. Tesla isn't profiling the road, and will drive off a cliff if the lines lead there, as the parent article points out. We had that figured out in the DARPA Grand Challenge a decade ago, where the vehicles didn't need road lines.

Compare Google's system.

[1] https://electrek.co/2016/06/26/tesla-autopilot-tests-pedestr...

Could a human have detected the vehicle? Our only sensor is line of sight optical (with a few mirrors) and near misses when merging are very common.

From reading the article, it sounds like the human did detect the vehicle, and that's what prevented a collision.

Yeah but it wasn't clear on the exact circumstances. If it was accelerating from behind then it could have been occluded by another car. The human may have only seen it because they were free from concentrating on other things, etc.

I like this video, which shows the Tesla autopilot in action with the driver-side display (which is usually hidden by the wheel) visible: https://www.youtube.com/watch?v=ZetTg73Ebyo

Looking at it, it seems that the autopilot mostly adds another layer of indirection. Instead of driving directly, the driver gives the autopilot orders like "maximum speed 60" or "move one lane left/right". The driver still has to pay attention to what's happening around to give it the correct orders. In the example given in the article, the driver still has to check whether the lane is free before giving the car an order to move into that lane.

To me this is analogous to streets that are dangerous because pedestrians and cyclists are seldom seen. Drivers come to expect very simple patterns, and roads are engineered with forgiving design, and so drivers know they can be distracted with no issues.

Then something changes.

I believe there will be another death in a Tesla vehicle with Autopilot enabled within a year. As drivers get more used to using it they will get more complacent. More features and improvements will roll out, requiring almost-but-not-quite complete inattention, which will increase driver's trust in the Autopilot, ironically putting them in more danger from the 1-in-100000 situations that humans handle easily.

Full autonomy is the only safe mode long term.

i agree. it's a dangerous game Tesla is playing at

Interesting that this story made the front page and the story today of the Tesla autopilot crash in China that was posted several times did not.

The scariest bit from the article: the driver who died was very familiar with the autopilot limitations.

Most people who die due to inattentive driving are aware of their car's limitations.

Now imagine a whole bunch of drivers who are not very familiar with autopilot limitations, especially if the autopilot works great 99.99% of the time.

if it works 99.99% of the time that will likely be higher than the average human driver

You're missing the point. The current Tesla "autopilot" is not ready, and is not intended, for full time driving. Not yet. However, it might appear to work well, and some Tesla drivers will forget that it's not ready. That's the problem with half-finished "autopilots".

I wonder to what extent the Tesla not seeing the SUV was due to it being gold colored. There must also be colors of cars that humans are less likely to see, too. I wonder if certain vehicle colors might be phased out to aid road safety in the near future.

I wrote the following a little while ago just after the auto pilot crash was reported. Since then my feelings on auto pilot haven't' changed. it's the future but right now it actually makes things harder due to the uncanny valley situation its currently in.....

If you've never used the Tesla autopilot its a weird feeling.

It's not a fully self driving car and its obviously not strictly human controlled. Unfortunately it ends up being more mentally taxing to use this hybrid approach than to just drive yourself.

Consider highway driving, with normal human powered mode you are in full control so if you see brake lights a half kilometer up ahead you can disengage the cruise control and react on your own, everyone who has driven is comfortable with this.

With this assisted driving the car doesn't slow down right away, and its not clear if the car just can't see the tail lights lighting up yet or if its decided it doesn't need to react yet and as such you start to second guess the car,

- should I drive or should I leave it to the car?

- what if it turns when I grab the wheel? Will I make things worse by driving?

- does the car even see the object up ahead? How can I tell, its impossible to expect the car to tell you of an object that it cant' even see.

It becomes just more taxing to use the hybrid approach to driving and as such I don't use it at all. I have no doubt that the Tesla auto pilot is safer than driving, but I also have no doubt that it can royally screw up.

I've come to the conclusion that some assisted driving, like auto braking for obstacles that you will imminently hit is good but the kind of assisted driving where it can almost automatically drive for you is not really anywhere near ready, if you follow Tesla's rules on how to use it, it actually makes driving harder:(

I have a Volvo XC90 which can drive itself to a certain extent (stays in its lane and can come to a complete stop) in traffic under 30 miles an hour and has dynamic cruise control for higher speed and I feel the same way.

Especially when breaking. The will it break or won't it (spoiler: it does) is nerve wracking.

> Especially when breaking. The will it break or won't it (spoiler: it does) is nerve wracking.

I would like to think that you intentionally spelled "braking" as "breaking" in a deliciously punny word play on the robustness of the technology itself, as well as its ability to stop a hurtling vehicle from 'breaking' itself against a stationary object.

Let's go with that ;)

I just did about 400 miles on autopilot today (and thousands previously) and I find it to be the total opposite. It's infinitely more relaxing than fully manual driving.

You still have to pay attention and decide, but the minutiae of making tiny steering movements is gone, leaving more capacity for higher level observation and planning.

The second guessing you describe simply doesn't happen for me. I'm capable of taking over and driving competently in any situation autopilot can handle. I never try to guess whether the car can see something; if it fails to respond in a way I like, I just take over. If I want to brake for a car and autopilot isn't braking, I brake. There's no need and no reason to wait for the car to maybe do it on its own or maybe not.

Disengaging autopilot, performing some maneuver, and reengaging it takes no significant time or effort beyond the maneuver itself. If you're sitting there wanting the car to react to something and it hasn't yet and you're waiting to see if it will, you're doing it all wrong.

> If you're sitting there wanting the car to react to something and it hasn't yet and you're waiting to see if it will, you're doing it all wrong.

Okay, but what if you're the sort of prematurely-conscientious person who holds a door for someone who's still 30 seconds away from the door? You'll always be doing things before the autopilot gets to them, because you're doing them before you're "supposed" to do them (which in normal human driving is almost never a problem, so there's nothing discouraging people from this.) Thus, the autopilot will never get to do anything, so there's no point to turning it on.

Indeed. If you're that way the system will probably not be useful to you. That's OK: most people aren't like that, and it doesn't have to be useful to 100% of the population.

Suppose you're right about that type of person and autopilot never getting to do anything. I think it's still worth having it on. People make errors. The one time that person misjudges something and is about to hit something on the road, there's a good chance autopilot will see it and brake. It only needs to happen once to be worth it.

Reply on news.ycombinator.com