Hacker News Reader

Comment by SilasX | Hacker News Reader

I would be a lot more forgiving of these screwups if Tesla didn't constantly swear up and down that they've solved self-driving cars.

As far back as 2016 they were claiming they had full SDC capability above human driver safety [1], and their recent Model Y announcement suggests that the only thing holding it up is regulatory approval, and not failure to achieve the desired spec.

>Model Y will have Full Self-Driving capability, enabling automatic driving on city streets and highways pending regulatory approval, as well as the ability to come find you anywhere in a parking lot.

[1] https://www.tesla.com/blog/all-tesla-cars-being-produced-now...

[2] https://news.ycombinator.com/item?id=19397942 (linking HN because it's hard to find the text in the page with their UI)

View on HN

I don't want to dismiss your whole point, because it is certainly valid, but that isn't really the issue here. It is entirely possible for bugs like this to exist in the self driving tech and for Tesla to be correct in their claims that Autopilot is on average safer than a human driver.

It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans. You and I certainly don't have enough data to say one way or another on that. I would bet even Tesla doesn't have enough data to say definitively. However writing this tech off as unsafe just because it makes what seems like an obvious mistake is a great way to slow progress which will result in more human deaths long term.

That’s one of those thorny AI-human problems, though — if I crash 1 in 10 drives, it’s my fault, and therefore much easier to rationalize to be, well, not my fault. If my Tesla crashes 1 in 100 drives, it’s a faulty machine and an obvious death trap.

It is a classic trolley problem. Is it worth killing a group of innocent people if it will save a larger group of innocent people? I can certainly see why people might be pushing for caution against Tesla since we don't know the size of either of those groups.

It could theoretically be a trolley problem. In practice Tesla are probably not that safe if you compare them driving themselves on freeways to humans driving under the same conditions.

I was going to point you to a study showing that autosteer reduced accidents by 40%, but Google told me that this was recently shown to be wrong. In fact it increased accidents by 60%.

See https://arstechnica.com/cars/2019/02/in-2017-the-feds-said-t... for details and verification.

The the paragraph directly after that 60% number:

>So does that mean that Autosteer actually makes crashes 59 percent more likely? Probably not. Those 5,714 vehicles represent only a small portion of Tesla's fleet, and there's no way to know if they're representative. And that's the point: it's reckless to try to draw conclusions from such flawed data. NHTSA should have either asked Tesla for more data or left that calculation out of its report entirely.

I will go back to my original statement at the start of this thread. No one in these comments has the data to say definitively whether Autopilot is safer than a human driver. I am skeptical that Tesla even has enough data for that. But I am also skeptical of people who take that unknown and the occasional anecdotal data point like the above video as proof that Autopilot is inherently less safe than humans.

Tesla turned over the best data set to the NHTSA. They had more data on Autopilot usage and chose not to turn it over. It's thus fair to assume that the data they did not provide would not have benefited the company. So if even the Tesla-provided data set shows Tesla is worse than a normal driver, then it's logical to assume that a full data set would show the same--or worse.

If you crash 1 in 10 drives you have no business being on the road. Most people can drive for years without even having a single, minor accident.

If Tesla crashes 1 in 100 drives, it's absolutely a death trap since that's still an obscenely bad accident rate.

No, it's still the issue. If there's a bug in the software code for a programmable pacemaker, it doesn't matter if the fundamental design is flawed. For governmental regulators, it's the actual implimentation and real world performance that matters.

Sure, a bug fix for a routine being called is maybe easier to fix, but the real world performance is still the important metric to track.

Ok, but a lot of people are arguing something along the lines of "there is a bug in the pacemaker, let's take take out everyone's pacemakers".

I am not saying we shouldn't be critical of Tesla or hold them accountable for their product. My point is simply that their product doesn't have to be infallible for it to still be an improvement over the current solution.

A pacemaker has objective improvements over the alternative, namely, death. So if a pacemaker has a bug, it's still overwhelmingly better than the baseline scenario of "no pacemakers."

A self-driving Tesla is not objectively better than a regular, human-driven car. The jury's still out on whether any self-driving car is even as good as the average human driver, so if one of them has a bug that causes serious accidents in reproducible situations, that's not better than the baseline situation it's being compared to.

> So if a pacemaker has a bug, it's still overwhelmingly better than the baseline scenario of "no pacemakers."

Unless it goes of when it's not needed and kills you.

Right, but the probability of you dying because you do have a pacemaker is (in the situations for which a pacemaker is prescribed) far less than the probability of you dying because you don't have a pacemaker.

The same cannot yet be said about "self-driving" cars.

> Right, but the probability of you dying because you do have a pacemaker is (in the situations for which a pacemaker is prescribed) far less than the probability of you dying because you don't have a pacemaker.

That assumes the pacemaker works more often than it doesn't. (Which is the case now.) It's an unstated assumption that doesn't always apply when generalizing your example.

No it doesn't assume that. We don't install pacemakers to everyone at birth.

Even if pacemaker fails 90% of the time 90% chance of death is better than a 100% without it.

You're assuming "fails" only accounts for false negatives. False positives are a thing. If there was a 90% chance of a pacemaker going off when it wasn't needed, they wouldn't be used as it'd cause nearly as meany deaths as it prevents, assuming no false negatives.

Another reason pacemakers are not comparable is that pacemakers do not present anything like the potential threat to third parties that self-driving cars do. Two reasons we can say that with confidence is that, with pacemakers, the scenarios are simpler and we have good statistical data.

>> Another reason pacemakers are not comparable is that pacemakers do not present anything like the potential threat to third parties that self-driving cars do.

Unless the person wearing pacemaker is driving a non-self-driving car.

As I wrote, specifically because I knew someone would make this reply, in this case we have adequate statistics to make a good estimate of the minuscule risk.

i would not take those chances. if the self driving car is only slightly better then average driver in fatal accidents that's not good enough. most of accidents happen because of distractions (texting and so on) and if a self driving car is only slightly better then average driver I can get a lot better if I just don't use my smartphone while driving and don't sit behind a wheel tired.

so it's not enough to be better then average driver. It needs to be as good as "good" driver or better before I would trust it.

You are setting up a false dichotomy. You don't have to choose between Autopilot and paying attention. The video above is an obvious example. It didn't result in an accident because the driver was paying attention and intervened.

Also you aren't considering that other drivers can be the cause of a potential accident. You can assume that the other drivers on the road are average drivers with all the distractions that come along with that. If you make the other drivers on average slightly safer, that improves your safety even if your behavior is completely unchanged.

> You are setting up a false dichotomy. You don't have to choose between Autopilot and paying attention. The video above is an obvious example. It didn't result in an accident because the driver was paying attention and intervened.

If you have to pay attention, then you might as well be driving.

This is just a silly argument to me. Would anyone suggest that cruise control is worthless because you still have to steer the car?

The more the car drives for you, the harder it is to pay attention and the slower the context switch when suddenly the situation demands it.

This is basic human nature.

Cruise control is useful to me because it means I won’t accidentally speed. But having to hold the wheel means I need to pay attention. If the car is driving itself human nature means you might not pay attention. Also if the car does something weird you have to decide in a split second if it is because of a hazard you haven’t seen but the car has, or because the software went wrong.

You do have to hold the wheel, precisely because you might find yourself having to assess what the car is doing "in a split second", and possibly issue corrective actions. You can't do that unless you are holding the wheel and are paying attention to what the car is doing! But all that means is that the car is not really driving itself; it is however dealing with the boring, predictable parts of the job and leaving the rest up to you - this makes it easier to be attentive, not harder. The tasks where it's hard to pay continued attention are ones where you have to do something that seems extremely predictable, but also has very rare events where it isn't. Computer assistance can actually help a lot with such cases.

That makes no sense. If I can’t pay attention to the boring predictable part of driving how am I going to pay attention when I’m not even the one driving?

As for the rare unpredictable stuff, I might not be able to predict, say, a ladder falling off a work truck, but I can recognize it could fall off, prepare for that type of event, probably even recognize it before the car does, and if I don’t die, I can learn from it.

I don’t really see how also having to worry about the car itself creating a rare event helps. Now I’ve got to spend additional time deciding whether to take over and if I do, the time I have to react is reduced.

The boring predictable parts of driving (lane keeping and speed keeping) are already basically unconscious muscle memory. It’s watching others for surprises that takes the mental energy.

Exactly - though speed keeping is nice for long road-trips simply not to get foot cramps.

Anyway, I'm yet to see a video of Tesla doing emergency breaks because of highway pileup where it's not only a couple of seconds after human driver could see a row of red breaking lights in front.

Well if marketing says you get a CNC mill and what you really get is three power feeds it's not worthless but it's still not what you were promised.

You definitely aren't doing your commute in a Tesla, then.

Not only that, this is a new type of driving where you have to actively pay attention and fight the vehicle.

i was talking about self driving cars, not really Tesla autopilot. but do you really think this (paying attention and correcting autopilot mistakes) would not be a bigger problem knowing well that no.1 reason of accidents is not paying attention.

yea, other drivers are also part of this, but even then I would like something alot better then average. sure there is a chance a distracted teenager is riding somewhere around you, but it's not better if there is now 10 slightly less distracted teenagers around you.

average is really not a great measure here if we are talking about self driving cars.

You’re implying that agency is the element that is most desired. Ie if a human crashes and kills themselves/others, it is ok, because someone is at fault.

If the car is anywhere from slightly to quite a lot safer, but accidents that result in injuries/deaths occur, then it is not ok.

Psychologically you may feel this to be ‘right’ but I would prefer a world with less injuries and deaths all round. And one day the courts will too

no I'm just saying that average is not good enough.

imagine lung cancer for example, if we take average then it's 6% risk taking smookers and not smookes, but if I decided not to smoke then it's 0.2%. so in this case I can do better for lowering the chances then when I would accept some other solution that would let us all have it at 5% witch might seem ok but not for me.

wich

I wouldn't trust my life to proprietary software, because its makers' motives obviously don't align with my own. If my safety was really paramount, they'd open their software to scrutiny.
It's unethical to leave life-and-death decisions to a black-box algorithm, when I know it was written for the primary purpose of gathering more money. Safety is a constraint, not their goal.

You already do that right now and basically have done it every day of your life.
The power grid, railway, traffic lights, elevators, etc. all these systems are critical and closed source, and you don't see them killing people on a wide scale nor even on a small scale but regular basis.

Yep. Add the code which controls ABS and ESP in modern cars.

Elevators I'll grant you, but the power grid, railway and traffic lights are all controlled by governments, councils and NGO-type organisations — they all have reliability as their primary concern, not churning out units for profit.
And it may be irrational, but wrongly-activating brakes feel like less of a risk than wrongly-activating steering or accelerators.
And anyway, “lots of things are untrustworthy” is not a good argument for trusting something else.

I think you're missing the forest for the trees. We're not even at the point where we're comparing safety rates. Can a Model Y drive through DC? Can it ignore traffic signals in favor of following hand signals from a uniformed traffic officer, or follow the directions of a construction worker having traffic in turns through a road that they've decided to turn into one lane? Can it deal with an unannounced presidential motorcade rerouting traffic? Because that happens literally every day in DC. If Tesla can't handle that, it's inaccurate to say it has "full SDC," even before you get into an analysis of safety.

It's because Tesla marketing material doesn't tell the whole truth, just the selected bit that sounds impressive. The conditions in which the Autopilot is safer than a human driver are so narrow and under so many restrictions that it's obvious it's less "self driving" and more "driver assist".
Indeed, I also find it disingenuous to claim the Autopilot is safer than a human driver even if it actually can't function at all in 95% of situations on the road. Guess saying "possibly safer than a human driver at these 3 things and only under these very specific conditions" isn't so catchy. Definitely not thousands-of-dollars catchy.

It's because -in general- marketing material doesn't tell the whole truth, just the selected bit that sounds impressive.

Which, unfortunately, doesn't excuse them when users take Tesla at their word that their Autopilot feature is fully self-driving and end up in an accident and/or dead.

Is this hyperbole, or do you think Tesla actually said that?

but still there's quite some difference in the fine prints between a new MacBook being "two times faster than previous model#" and a Tesla "comes with full self driving hardware&"
# only on specific workloads
& don't ever get distracted because it can literally kill you

this is the point I see getting lost. This isn't someone overstating the how good a pair of pants make me look. It's more like selling a flame thrower as a weeding tool.
I know that sometimes these lines are objective, but acting like you can't tell okay and not okay apart because the line is gray in some cases just seems to be bad faith.
If you are okay with Tesla misleading selling a prototype feature in the name of disruption, own your argument. If this level of risk is just okay to you because of the potential long term upsides...I disagree but I respect your willingness to say it. Just don't make a different one because you are afraid of the social consequences of your actual point.

Don't you mean selling a weeding tool as a "flame thrower" ?

What you describe would be a disappointment. What he describe would be life-threatening.

Thank you for the assist.

Flamethrowers are regularly used as weeding tools though. The best way to kill weeds on the edge of a field or a ditch is to burn them.

Tesla did actually sell a propane torch that's a weeding tool (or asphalt surfacing tool, paint stripping tool etc) and they called it a flamethrower

Yeah and the auto dealer industry is notoriously deceptive too. Telsa told us they'd be different. Well they're not selling cars through dealers, that much is true, but it seems clear the deception is still in force.

That's probably very true but this isn't about the dealer as much as the manufacturer. When one says they have 4 crumple zones to absorb the energy of an impact and you later find out that the passengers were those crumple zones, someone goes to prison. When Tesla claims that their system is fully self driving at a level safer than a human driver they're doing the same. Because they know very well that system is just a glorified driver assist that can do any kind of driving only in the most narrow and perfect of conditions. Take it out of that narrow Autopilot "comfort zone" and you have plain old driver assist.

As far as I'm aware, Tesla has never claimed their cars will be able to handle all those situations. "Self driving car" is a vague term that has different definitions depending on who you ask. You are considering "self driving" as level 5 autonomy and Tesla probably considers it somewhere between 3-4 autonomy. I don't think it makes sense to get angry because their definition of the phrase is not the same as your definition of the phrase.

Tesla skirts morality.
The feature is called 'Full self driving'. Their sales reps regularly told customers to take their hands off the steering wheel of their "Autopilot". Sure, the fine print says "always pay attention", but there's an entire marketing scheme going on here which is borderline dishonest.

> "Sure, the fine print says "always pay attention","
In fact the not-so-fine print in the user manual says to keep your hands on the wheels, but the promotional material tells a different story, as does Musk when he takes his hands off the wheel on national television.

Honestly, I'm surprised a false advertising case hasn't been pursued if this is the case. Marketing practices this inaccurate are well outside the puffery defense.

That’s just called “driving” and if the car can’t handle those situations it’s not a “self driving car.” It’s just a driver assistance feature.

From what I understand, it can't even negotiate a traffic light.
Like here's a recent article swooning over it coming out sometime soon:
https://electrek.co/2018/12/09/tesla-autopilot-soon-traffic-...
The article there also says that the more capable version will require new hardware, which isn't something they have been admitting very readily over the years.

Furthermore, the demarcation between what it can handle and what it cannot is neither simple nor clear. As this case shows, you can be bowling down the sort of highway it is supposedly suitable for, yet suddenly come across something it cannot deal with - and maybe something it did negotiate successfully in the past.
As long as Tesla says that driver vigilance is required at all times, while simultaneously promoting it as if it were true automation, the risk to third parties, from Tesla autopilot users who don't understand what's going on here, is unacceptable.

> It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans. You and I certainly don't have enough data to say one way or another on that. I would bet even Tesla doesn't have enough data to say definitively. However writing this tech off as unsafe just because it makes what seems like an obvious mistake is a great way to slow progress which will result in more human deaths long term.
And where is the research to show that? To prove that within any significance, they would have to have millions and millions of miles to compare against human drivers. This is new technology and making false claims about its safety is dangerous.
We as a society force drug companies to rigorously show that their drugs do what they say they do and all of their side effects are properly accounted for. This should be the case here too.

The last time anyone tried to make that case statistically, it turned out to be bogus [1].
Bizarrely, this report was from the NHTSA, raising the sort of regulatory capture issues that have surfaced between Boeing and the FAA.
Also, when such a study is performed, we must be careful that technologies are not conflated to claim more than properly can be. The effectiveness of less powerful technologies such as lane-keeping assist and automatic emergency braking tells us nothing about the safety of full-authority automated driving.
[1] https://arstechnica.com/cars/2019/02/in-2017-the-feds-said-t...

Hang on.
Unless something has changed, the Telsa self driving tech intentionally ignores stationary objects.
And Telsa are claiming this is a feature not a bug.
Am I understanding this correctly?
I don't know anyone who isn't suicidal who intentionally ignores stationary objects while driving.

Do you have anything to back this up? That seems like a pretty big claim to make unsubstantiated. If autopilot literally ignores staionary objects, you would see them crash much more often than they do.

It has been in the media.
https://www.google.com/search?q=tesla+ignores+stationary+obj...
Edit to add: apologies for terse response, a bit distracted at present

We shouldn't have to conjecture about this. We should be able to read and understand the algorithm.

Is it, though? If every car that drove that route was a Tesla on autopilot, you'd probably have more traffic fatalities than the national total, and that doesn't even factor in every other similar traffic barrier this can occur at.

> see self driving cars run into solid and stationary objects, but human drivers do that all the time too.
Human drivers who are distracted do that all the time, AP is supposed to avoid that, it is supposed to be alert all the time, but when it sees a stationary object, the result of their algo is, "it must be a sign we can somehow go through"
We need to make it very clear that self driving cars are better than driving drunk, or when you haven't slept in 24+ hours, but if you are a driver who pays attention, don't use this tech.
And it is not that I don't want the tech to take over the world, I wish I could just put my kids in a self driving car and have the car take them to the school that is 4 miles from home, but we are nowhere close to that, even with me in the driver seat, if I only have seconds to take over before I end up on a ditch or worse.

There are currently no self driving cars which a drunk or exhausted person could operate with an acceptable level of safety.

It’s far from clear in the statistics that self driving cars today (without the use of a safety driver) are safer than driving drunk. IMO Tesla autopilot, fairly clearly is less safe than driving drunk if a human is not carefully monitoring.

I think this graph:
https://upload.wikimedia.org/wikipedia/commons/a/a5/Causes_o...
puts a lot of arguments (and young people) to rest.

Not at all. It says that car accidents are a problem. It does not say that the answer is to pretend your driver assistance tech is actually self-driving tech.

I was addressing the point in the parent comment:
> The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans.

It may be better than humans on average, but is it safer than a sober person who is driving responsibly in a similarly priced vehicle? Anecdotally, most accidents I've seen have been caused by at least one driver doing some stupid shit. Based on my own personal experiences, I'm just not comfortable surrending control of my vehicle. Especially not when we could be comparing apples to oranges. Safety of a self driving system should be evaluated against currently available safety technologies like blind spot detection etc. If you're including older vehicles that don't have the best possible non-self driving technology in your data, you're making a comparison that is inherently biased in favor of self driving cars.
All that being said, I do think self driving is what we will inevitably arrive at...we just need to have a higher degree of confidence in it before it's widely used in my opinion.

Nn tech debt.

Model Y will have Full Self-Driving capability, enabling automatic driving on city streets and highways pending regulatory approval, as well as the ability to come find you anywhere in a parking lot.
We've heard that before, for the Model X.

> As far back as 2016 they were claiming they had full SDC capability above human driver safety
One crash/incident doesn’t mean it’s less safe than a human driver on average, even if it’s something a human driver might have avoided.
We should expect different failure modes from a machine, but adopt it anyway if it avoids enough human failure modes to make up for it.
Source: my thermostat thought 33C was an acceptable temp once, but I still didn’t switch to full-manual HVAC control.

Don't Teslas fare significantly worse than cars in similar class in terms of driver fatalities?

I don't have the fatalities stats, but their safety ratings have been perfect.
[0] https://www.cnbc.com/2018/09/20/tesla-model-3-earns-perfect-...

Dunno, maybe. I do wonder which cars Tesla gets lumped in with.
There can be a difference between a $75k ICE driver and a $75k electric driver.
The point is, one stupid decision by a machine doesn’t prove much.

So... Two planes falling out of the sky, killing 400 shouldn't be taken as proving anything?
Or how about 1 stupid decision by a radiotherapy machine? Ever heard of THERAC-25?
In each case, it was just 1 stupid decision by a machine.
The entire reason Engineering as a practice is a thing is because when you implement the capacity for a stupid decision into a system that is then mass produced, dire consequences can result.
I look down on any thought process that doesn't discriminate between the difference between 0 and 1.
If the system provably worked, that decision would not have happened (0). The stupid decision happened (1), however, which means it can happen again at a poorly understood confluence of circumstances.
To err is human, and we forgive each other every day for it.
To err as a machine is a condemnation to the refuse bucket, repair shop, or back to the drawing board.
To err so egregiously as a machine to cause an operator and those around them to lose their life is willful and moral negligence on the part of the system's designer. Slack is cut when good faith is demonstrated, but liability is unambiguous. The hazard would not be there if you hadn't put it there.

And people have died and been paralyzed as a direct result of the flu vaccine. Death and paralyses that would not have occurred had they not received that vaccine.
Does that mean the flu vaccine should get dumped in the refuse bin?
Similarly, some other aircraft flying today/tomorrow has automation with an unknown bug/issue that will cause loss of life. Should we disable everything except the 6-pack and stick and rudder?
It would have saved the lives killed by automation, but we would have more aviation death overall.

Does that mean you believe the decision to ground 737 max 8s is incorrect?

Liability is still clear, and in the specific case of medical practice, a degree of "we can't foresee everything" is implicit in that our understanding of the governing principles of the human body is incomplete.
Aviation does not have that excuse. The 737 MAX 8 system description is enumerated from the ground up. Seeing as there was so much recertification effort that didn't need to be done, it makes the failure to properly handle the MCAS implementation all the more damning.
This wasn't some subtle bug. This was an outright terrible design choice. Anyone with any experience composing complex systems out of smaller functional building blocks should have been able to look at the outputs, look at the inputs, and realize there was the potential for catastrophic malfunction.
As I've said elsewhere, automation should make flying a plane easier when functional. When non-functional, however, the pilot should still be able to salvage the plane. That requires clear communication of what automation does, and what it's failure modes are.

Are you asking about Teslas or Teslas being driven by Autopilot? Autopilot so far has a fatality rate of roughly 0.25 per 100 million miles. Just for information's sake the overall rate in the US in 2017 was 1.16 per 100 million miles. Although there are plenty of caveats that should prevent you from comparing those numbers directly. The national number is for all cars, all drivers, all conditions, etc while those driving a Tesla in Autopilot are generally considered to be in a safer cars, safer conditions, and be safer drivers than average.

Tesla drivers have the luxury of turning autopilot off in suboptimal conditions. Human drivers don't.
Like you say, this number is really valuable only for Tesla marketing.

Tesla would have had 0 fatalities in total if the Autopilot disengaged just before hitting the barrier.

For all driving protocols P, and accidents A, P would have zero fatalities if it switched to a better protocol just before encountering the situation that led to A.

That's exactly what I am trying to say. Yet Tesla implies their tech is statistically superior to human drivers, yet can disengage at any time.

Ah okay. Sorry, easy to misread people on this topic.

That stat of 1.16 includes motor vehicles, motorcycles, pedestrians, buses, bicyclists and trucks. How is this a comparison to luxury electric cars again?

They solved it, they just need 100x faster GPUs in cars to be reliable outside datacenters ;-)

The passage says “will” which implies the future.

Reply on news.ycombinator.com