Hacker News Reader

Comment by slg | Hacker News Reader

I don't want to dismiss your whole point, because it is certainly valid, but that isn't really the issue here. It is entirely possible for bugs like this to exist in the self driving tech and for Tesla to be correct in their claims that Autopilot is on average safer than a human driver.

It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans. You and I certainly don't have enough data to say one way or another on that. I would bet even Tesla doesn't have enough data to say definitively. However writing this tech off as unsafe just because it makes what seems like an obvious mistake is a great way to slow progress which will result in more human deaths long term.

View on HN

I think you're missing the forest for the trees. We're not even at the point where we're comparing safety rates. Can a Model Y drive through DC? Can it ignore traffic signals in favor of following hand signals from a uniformed traffic officer, or follow the directions of a construction worker having traffic in turns through a road that they've decided to turn into one lane? Can it deal with an unannounced presidential motorcade rerouting traffic? Because that happens literally every day in DC. If Tesla can't handle that, it's inaccurate to say it has "full SDC," even before you get into an analysis of safety.

It's because Tesla marketing material doesn't tell the whole truth, just the selected bit that sounds impressive. The conditions in which the Autopilot is safer than a human driver are so narrow and under so many restrictions that it's obvious it's less "self driving" and more "driver assist".

Indeed, I also find it disingenuous to claim the Autopilot is safer than a human driver even if it actually can't function at all in 95% of situations on the road. Guess saying "possibly safer than a human driver at these 3 things and only under these very specific conditions" isn't so catchy. Definitely not thousands-of-dollars catchy.

It's because -in general- marketing material doesn't tell the whole truth, just the selected bit that sounds impressive.

but still there's quite some difference in the fine prints between a new MacBook being "two times faster than previous model#" and a Tesla "comes with full self driving hardware&"

# only on specific workloads

& don't ever get distracted because it can literally kill you

this is the point I see getting lost. This isn't someone overstating the how good a pair of pants make me look. It's more like selling a flame thrower as a weeding tool.

I know that sometimes these lines are objective, but acting like you can't tell okay and not okay apart because the line is gray in some cases just seems to be bad faith.

If you are okay with Tesla misleading selling a prototype feature in the name of disruption, own your argument. If this level of risk is just okay to you because of the potential long term upsides...I disagree but I respect your willingness to say it. Just don't make a different one because you are afraid of the social consequences of your actual point.

Don't you mean selling a weeding tool as a "flame thrower" ?

What you describe would be a disappointment. What he describe would be life-threatening.

Tesla did actually sell a propane torch that's a weeding tool (or asphalt surfacing tool, paint stripping tool etc) and they called it a flamethrower

Flamethrowers are regularly used as weeding tools though. The best way to kill weeds on the edge of a field or a ditch is to burn them.

Thank you for the assist.

Which, unfortunately, doesn't excuse them when users take Tesla at their word that their Autopilot feature is fully self-driving and end up in an accident and/or dead.

Is this hyperbole, or do you think Tesla actually said that?

Yeah and the auto dealer industry is notoriously deceptive too. Telsa told us they'd be different. Well they're not selling cars through dealers, that much is true, but it seems clear the deception is still in force.

That's probably very true but this isn't about the dealer as much as the manufacturer. When one says they have 4 crumple zones to absorb the energy of an impact and you later find out that the passengers were those crumple zones, someone goes to prison. When Tesla claims that their system is fully self driving at a level safer than a human driver they're doing the same. Because they know very well that system is just a glorified driver assist that can do any kind of driving only in the most narrow and perfect of conditions. Take it out of that narrow Autopilot "comfort zone" and you have plain old driver assist.

From what I understand, it can't even negotiate a traffic light.

Like here's a recent article swooning over it coming out sometime soon:

https://electrek.co/2018/12/09/tesla-autopilot-soon-traffic-...

The article there also says that the more capable version will require new hardware, which isn't something they have been admitting very readily over the years.

Furthermore, the demarcation between what it can handle and what it cannot is neither simple nor clear. As this case shows, you can be bowling down the sort of highway it is supposedly suitable for, yet suddenly come across something it cannot deal with - and maybe something it did negotiate successfully in the past.

As long as Tesla says that driver vigilance is required at all times, while simultaneously promoting it as if it were true automation, the risk to third parties, from Tesla autopilot users who don't understand what's going on here, is unacceptable.

As far as I'm aware, Tesla has never claimed their cars will be able to handle all those situations. "Self driving car" is a vague term that has different definitions depending on who you ask. You are considering "self driving" as level 5 autonomy and Tesla probably considers it somewhere between 3-4 autonomy. I don't think it makes sense to get angry because their definition of the phrase is not the same as your definition of the phrase.

Tesla skirts morality.

The feature is called 'Full self driving'. Their sales reps regularly told customers to take their hands off the steering wheel of their "Autopilot". Sure, the fine print says "always pay attention", but there's an entire marketing scheme going on here which is borderline dishonest.

> "Sure, the fine print says "always pay attention","

In fact the not-so-fine print in the user manual says to keep your hands on the wheels, but the promotional material tells a different story, as does Musk when he takes his hands off the wheel on national television.

Honestly, I'm surprised a false advertising case hasn't been pursued if this is the case. Marketing practices this inaccurate are well outside the puffery defense.

That’s just called “driving” and if the car can’t handle those situations it’s not a “self driving car.” It’s just a driver assistance feature.

> It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans. You and I certainly don't have enough data to say one way or another on that. I would bet even Tesla doesn't have enough data to say definitively. However writing this tech off as unsafe just because it makes what seems like an obvious mistake is a great way to slow progress which will result in more human deaths long term.

And where is the research to show that? To prove that within any significance, they would have to have millions and millions of miles to compare against human drivers. This is new technology and making false claims about its safety is dangerous.

We as a society force drug companies to rigorously show that their drugs do what they say they do and all of their side effects are properly accounted for. This should be the case here too.

The last time anyone tried to make that case statistically, it turned out to be bogus [1].

Bizarrely, this report was from the NHTSA, raising the sort of regulatory capture issues that have surfaced between Boeing and the FAA.

Also, when such a study is performed, we must be careful that technologies are not conflated to claim more than properly can be. The effectiveness of less powerful technologies such as lane-keeping assist and automatic emergency braking tells us nothing about the safety of full-authority automated driving.

[1] https://arstechnica.com/cars/2019/02/in-2017-the-feds-said-t...

> see self driving cars run into solid and stationary objects, but human drivers do that all the time too.

Human drivers who are distracted do that all the time, AP is supposed to avoid that, it is supposed to be alert all the time, but when it sees a stationary object, the result of their algo is, "it must be a sign we can somehow go through"

We need to make it very clear that self driving cars are better than driving drunk, or when you haven't slept in 24+ hours, but if you are a driver who pays attention, don't use this tech.

And it is not that I don't want the tech to take over the world, I wish I could just put my kids in a self driving car and have the car take them to the school that is 4 miles from home, but we are nowhere close to that, even with me in the driver seat, if I only have seconds to take over before I end up on a ditch or worse.

There are currently no self driving cars which a drunk or exhausted person could operate with an acceptable level of safety.

It’s far from clear in the statistics that self driving cars today (without the use of a safety driver) are safer than driving drunk. IMO Tesla autopilot, fairly clearly is less safe than driving drunk if a human is not carefully monitoring.

No, it's still the issue. If there's a bug in the software code for a programmable pacemaker, it doesn't matter if the fundamental design is flawed. For governmental regulators, it's the actual implimentation and real world performance that matters.

Sure, a bug fix for a routine being called is maybe easier to fix, but the real world performance is still the important metric to track.

Ok, but a lot of people are arguing something along the lines of "there is a bug in the pacemaker, let's take take out everyone's pacemakers".

I am not saying we shouldn't be critical of Tesla or hold them accountable for their product. My point is simply that their product doesn't have to be infallible for it to still be an improvement over the current solution.

A pacemaker has objective improvements over the alternative, namely, death. So if a pacemaker has a bug, it's still overwhelmingly better than the baseline scenario of "no pacemakers."

A self-driving Tesla is not objectively better than a regular, human-driven car. The jury's still out on whether any self-driving car is even as good as the average human driver, so if one of them has a bug that causes serious accidents in reproducible situations, that's not better than the baseline situation it's being compared to.

> So if a pacemaker has a bug, it's still overwhelmingly better than the baseline scenario of "no pacemakers."

Unless it goes of when it's not needed and kills you.

Right, but the probability of you dying because you do have a pacemaker is (in the situations for which a pacemaker is prescribed) far less than the probability of you dying because you don't have a pacemaker.

The same cannot yet be said about "self-driving" cars.

> Right, but the probability of you dying because you do have a pacemaker is (in the situations for which a pacemaker is prescribed) far less than the probability of you dying because you don't have a pacemaker.

That assumes the pacemaker works more often than it doesn't. (Which is the case now.) It's an unstated assumption that doesn't always apply when generalizing your example.

No it doesn't assume that. We don't install pacemakers to everyone at birth.

Even if pacemaker fails 90% of the time 90% chance of death is better than a 100% without it.

You're assuming "fails" only accounts for false negatives. False positives are a thing. If there was a 90% chance of a pacemaker going off when it wasn't needed, they wouldn't be used as it'd cause nearly as meany deaths as it prevents, assuming no false negatives.

Another reason pacemakers are not comparable is that pacemakers do not present anything like the potential threat to third parties that self-driving cars do. Two reasons we can say that with confidence is that, with pacemakers, the scenarios are simpler and we have good statistical data.

>> Another reason pacemakers are not comparable is that pacemakers do not present anything like the potential threat to third parties that self-driving cars do.

Unless the person wearing pacemaker is driving a non-self-driving car.

As I wrote, specifically because I knew someone would make this reply, in this case we have adequate statistics to make a good estimate of the minuscule risk.

i would not take those chances. if the self driving car is only slightly better then average driver in fatal accidents that's not good enough. most of accidents happen because of distractions (texting and so on) and if a self driving car is only slightly better then average driver I can get a lot better if I just don't use my smartphone while driving and don't sit behind a wheel tired.

so it's not enough to be better then average driver. It needs to be as good as "good" driver or better before I would trust it.

I wouldn't trust my life to proprietary software, because its makers' motives obviously don't align with my own. If my safety was really paramount, they'd open their software to scrutiny.

It's unethical to leave life-and-death decisions to a black-box algorithm, when I know it was written for the primary purpose of gathering more money. Safety is a constraint, not their goal.

You already do that right now and basically have done it every day of your life.

The power grid, railway, traffic lights, elevators, etc. all these systems are critical and closed source, and you don't see them killing people on a wide scale nor even on a small scale but regular basis.

Elevators I'll grant you, but the power grid, railway and traffic lights are all controlled by governments, councils and NGO-type organisations — they all have reliability as their primary concern, not churning out units for profit.

And it may be irrational, but wrongly-activating brakes feel like less of a risk than wrongly-activating steering or accelerators.

And anyway, “lots of things are untrustworthy” is not a good argument for trusting something else.

Yep. Add the code which controls ABS and ESP in modern cars.

You’re implying that agency is the element that is most desired. Ie if a human crashes and kills themselves/others, it is ok, because someone is at fault.

If the car is anywhere from slightly to quite a lot safer, but accidents that result in injuries/deaths occur, then it is not ok.

Psychologically you may feel this to be ‘right’ but I would prefer a world with less injuries and deaths all round. And one day the courts will too

no I'm just saying that average is not good enough.

imagine lung cancer for example, if we take average then it's 6% risk taking smookers and not smookes, but if I decided not to smoke then it's 0.2%. so in this case I can do better for lowering the chances then when I would accept some other solution that would let us all have it at 5% witch might seem ok but not for me.

wich

You are setting up a false dichotomy. You don't have to choose between Autopilot and paying attention. The video above is an obvious example. It didn't result in an accident because the driver was paying attention and intervened.
Also you aren't considering that other drivers can be the cause of a potential accident. You can assume that the other drivers on the road are average drivers with all the distractions that come along with that. If you make the other drivers on average slightly safer, that improves your safety even if your behavior is completely unchanged.

> You are setting up a false dichotomy. You don't have to choose between Autopilot and paying attention. The video above is an obvious example. It didn't result in an accident because the driver was paying attention and intervened.
If you have to pay attention, then you might as well be driving.

Not only that, this is a new type of driving where you have to actively pay attention and fight the vehicle.

You definitely aren't doing your commute in a Tesla, then.

This is just a silly argument to me. Would anyone suggest that cruise control is worthless because you still have to steer the car?

The more the car drives for you, the harder it is to pay attention and the slower the context switch when suddenly the situation demands it.
This is basic human nature.

Cruise control is useful to me because it means I won’t accidentally speed. But having to hold the wheel means I need to pay attention. If the car is driving itself human nature means you might not pay attention. Also if the car does something weird you have to decide in a split second if it is because of a hazard you haven’t seen but the car has, or because the software went wrong.

You do have to hold the wheel, precisely because you might find yourself having to assess what the car is doing "in a split second", and possibly issue corrective actions. You can't do that unless you are holding the wheel and are paying attention to what the car is doing! But all that means is that the car is not really driving itself; it is however dealing with the boring, predictable parts of the job and leaving the rest up to you - this makes it easier to be attentive, not harder. The tasks where it's hard to pay continued attention are ones where you have to do something that seems extremely predictable, but also has very rare events where it isn't. Computer assistance can actually help a lot with such cases.

The boring predictable parts of driving (lane keeping and speed keeping) are already basically unconscious muscle memory. It’s watching others for surprises that takes the mental energy.

Exactly - though speed keeping is nice for long road-trips simply not to get foot cramps.
Anyway, I'm yet to see a video of Tesla doing emergency breaks because of highway pileup where it's not only a couple of seconds after human driver could see a row of red breaking lights in front.

That makes no sense. If I can’t pay attention to the boring predictable part of driving how am I going to pay attention when I’m not even the one driving?
As for the rare unpredictable stuff, I might not be able to predict, say, a ladder falling off a work truck, but I can recognize it could fall off, prepare for that type of event, probably even recognize it before the car does, and if I don’t die, I can learn from it.
I don’t really see how also having to worry about the car itself creating a rare event helps. Now I’ve got to spend additional time deciding whether to take over and if I do, the time I have to react is reduced.

Well if marketing says you get a CNC mill and what you really get is three power feeds it's not worthless but it's still not what you were promised.

i was talking about self driving cars, not really Tesla autopilot. but do you really think this (paying attention and correcting autopilot mistakes) would not be a bigger problem knowing well that no.1 reason of accidents is not paying attention.
yea, other drivers are also part of this, but even then I would like something alot better then average. sure there is a chance a distracted teenager is riding somewhere around you, but it's not better if there is now 10 slightly less distracted teenagers around you.
average is really not a great measure here if we are talking about self driving cars.

It may be better than humans on average, but is it safer than a sober person who is driving responsibly in a similarly priced vehicle? Anecdotally, most accidents I've seen have been caused by at least one driver doing some stupid shit. Based on my own personal experiences, I'm just not comfortable surrending control of my vehicle. Especially not when we could be comparing apples to oranges. Safety of a self driving system should be evaluated against currently available safety technologies like blind spot detection etc. If you're including older vehicles that don't have the best possible non-self driving technology in your data, you're making a comparison that is inherently biased in favor of self driving cars.
All that being said, I do think self driving is what we will inevitably arrive at...we just need to have a higher degree of confidence in it before it's widely used in my opinion.

Hang on.
Unless something has changed, the Telsa self driving tech intentionally ignores stationary objects.
And Telsa are claiming this is a feature not a bug.
Am I understanding this correctly?
I don't know anyone who isn't suicidal who intentionally ignores stationary objects while driving.

Do you have anything to back this up? That seems like a pretty big claim to make unsubstantiated. If autopilot literally ignores staionary objects, you would see them crash much more often than they do.

It has been in the media.
https://www.google.com/search?q=tesla+ignores+stationary+obj...
Edit to add: apologies for terse response, a bit distracted at present

We shouldn't have to conjecture about this. We should be able to read and understand the algorithm.

> It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans
I fail to understand this line of reasoning. Are you saying because humans tend to run into solid objects it is okay if the self driving cars do it, only if it does less times than humans?

I think this graph:
https://upload.wikimedia.org/wikipedia/commons/a/a5/Causes_o...
puts a lot of arguments (and young people) to rest.

Not at all. It says that car accidents are a problem. It does not say that the answer is to pretend your driver assistance tech is actually self-driving tech.

I was addressing the point in the parent comment:
> The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans.

Is it, though? If every car that drove that route was a Tesla on autopilot, you'd probably have more traffic fatalities than the national total, and that doesn't even factor in every other similar traffic barrier this can occur at.

Nn tech debt.

That’s one of those thorny AI-human problems, though — if I crash 1 in 10 drives, it’s my fault, and therefore much easier to rationalize to be, well, not my fault. If my Tesla crashes 1 in 100 drives, it’s a faulty machine and an obvious death trap.

If you crash 1 in 10 drives you have no business being on the road. Most people can drive for years without even having a single, minor accident.
If Tesla crashes 1 in 100 drives, it's absolutely a death trap since that's still an obscenely bad accident rate.

It is a classic trolley problem. Is it worth killing a group of innocent people if it will save a larger group of innocent people? I can certainly see why people might be pushing for caution against Tesla since we don't know the size of either of those groups.

It could theoretically be a trolley problem. In practice Tesla are probably not that safe if you compare them driving themselves on freeways to humans driving under the same conditions.

I was going to point you to a study showing that autosteer reduced accidents by 40%, but Google told me that this was recently shown to be wrong. In fact it increased accidents by 60%.
See https://arstechnica.com/cars/2019/02/in-2017-the-feds-said-t... for details and verification.

The the paragraph directly after that 60% number:
>So does that mean that Autosteer actually makes crashes 59 percent more likely? Probably not. Those 5,714 vehicles represent only a small portion of Tesla's fleet, and there's no way to know if they're representative. And that's the point: it's reckless to try to draw conclusions from such flawed data. NHTSA should have either asked Tesla for more data or left that calculation out of its report entirely.
I will go back to my original statement at the start of this thread. No one in these comments has the data to say definitively whether Autopilot is safer than a human driver. I am skeptical that Tesla even has enough data for that. But I am also skeptical of people who take that unknown and the occasional anecdotal data point like the above video as proof that Autopilot is inherently less safe than humans.

Tesla turned over the best data set to the NHTSA. They had more data on Autopilot usage and chose not to turn it over. It's thus fair to assume that the data they did not provide would not have benefited the company. So if even the Tesla-provided data set shows Tesla is worse than a normal driver, then it's logical to assume that a full data set would show the same--or worse.

Reply on news.ycombinator.com