Hacker News Reader

Comment by nostrademons | Hacker News Reader

"Move fast and break things."

I really don't understand how a company can get away with using its customers as QA when they sell products that can kill people. It's one thing if it's a website for throwing virtual sheep at another; nobody really cares if your Facebook feed omits an update or two. It's another thing if a software bug means that you die. Or at least it should be - I'm amazed at the number of people who seemingly don't care.

I also wonder what this is doing to Tesla's brand reputation. I was just in the market for solar panels and Tesla's offering (both the panels and the solar roof) was very, very attractive on price & aesthetics, but their reliability record with their cars made me think twice about buying a 25+ year product that keeps my home powered and dry.

View on HN

> I really don't understand how a company can get away with using its customers as QA when they sell products that can kill people

There's a term for it: "normalization of deviance"[1]. A highly risky thing is done and doesn't fail as early as thought. The people doing it assume that means the risk is lower than they thought, but it's not, and they just happened to get lucky.

I'm sure Elon knows this, so probably believes their tech can advance enough before their luck turns. I personally think he's deadly wrong on this, but it seems like a lot of Silicon Valley Tesla owners who sleep or read their phone while on Autopilot accept it at face value.

[1] - https://en.wikipedia.org/wiki/Normalization_of_deviance

> There's a term for it: "normalization of deviance"

I believe this is not entirely correct, as the parent talks about how a company can get away with something, while the term you use applies to the company itself.

The main question should be: why do we (the government) even allow companies to do something like this?

Say his name, Walter Huang

Obviously I have no idea what's in their heads, but a plausible train of thought goes like:

Legal : If you sell it as a drive assist and not a driver replacement, you're off the hook, with a high probability. It's on the drivers if they get complacent. There may even be precedents with similar products for trucks.

Moral: pushing this now will very likely kill some people that would otherwise be alive (drivers or pedestrians). But it's very unlikely to be a carnage - I'd dare say even impossible. On the other hand, moving the technology forward and directly into widespread commercial use will save people, and likely even... what's the equivalent word for a "carnage" but where people are saved? Miracle.

Branding: well, there you have a real risk. Public opinion is fickle, and a particularly gruesome accident might do some real damage, both to Tesla and to self driving technologies in general. But then, they do have a high risk high reward mentality, and the chance is good that the first bad accident (statistically inevitable) will come after the advantages are proven and well established.

> On the other hand, moving the technology forward and directly into widespread commercial use will save people

This is the only rationale I can come up with for Tesla's approach. The tenets go like this:

1) The ultimate realization of self driving will be successful, and will have profoundly positive impacts on society (e.g. safety, urban planning, ease and availability of transportation, economic)

2) The path to that ultimate realization contains many local optima traps

3) The only way to push past those local optima is by acquiring large amount of real world automated driving data, using systems in production

From a technical standpoint, I'm disinclined to disagree with them on the necessity of large amounts of real world system behavior data as a key ingredient.

Personally, I also think we (humans) have become way too culturally risk adverse. Or, to put the calculus another way: if I have a credible belief that I can save some number of future humans, what is the acceptable moral number of current humans to place at risk and/or kill?

Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."

From a purely ethical standpoint (leaving legal aside), the crux would seem to be informed users. Not all Tesla drivers are likely capable of understanding the current SotA, and I'd hope (and expect) that Tesla's phased rollout of the beta is informed by previous user interaction with assisted driving.

> if I have a credible belief that I can save some number of future humans, what is the acceptable moral number of current humans to place at risk and/or kill?

> Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."

I think that what we've learned throughout human history is that no one should be trusted and given the power to make such decisions. It has invariably led to disasters. The very idea that someone can decide to put someone else's life in danger for potential future advances gives too many perverse incentives - chief of which, rationalization that "more people will be saved" even as the experiment crashes and burns.

Bullshit. We've done that countless times, to good effect.

When it's for the greater good, or by the winners, we conveniently omit it from history.

When it's for the greater evil, or by the losers, we publicize it as a cautionary tale of moral hubris.

War aside, what’s an example of innocent lives being sacrificed for some sort of greater good that got omitted from official history?

>> War aside, what’s an example of innocent lives being sacrificed for some sort of greater good that got omitted from official history?

Example 1: Vaccines

This is an interesting example because the winners and losers seem to be proportionately spread across almost all of society (except anti-Vaxxers who benefit from herd immunity but don't sacrifice with the minute risk of vaccines). See also: https://www.hrsa.gov/vaccine-compensation/index.html

Example 2: War on Crime / Safety

Another example would be the war on crime. Liberties and lives are sacrificed in the name of safety for others. Unfortunately, the liberty/lives sacrificed fall disproportionately into one group, and those protected fall disproportionately into a different group group. If you look at the 2nd order effects (impoverishment, single-parent households, effects on education and future) lots more lives are lost, again in a subset of society.

This is of course highly controversial, but deaths in revolutions or shortly after that led to eventually much better living conditions might be considered, depending on your political views, to be such a case.

Cars. Cars kill hundreds of thousands people each year. But we accept it, for the greater good of modern civilization.

That's not an example of putting this kind of decision power in the hands of one person. It is a complicated history that likely started with almost no fatalities, and steadily increased as speeds increased, with no clear point where a line was drawn.

Not omitted, but the deaths and/or illnesses resulting from drug trials are almost never widely publicized. Conversely, any improvement brought by a new drug will be drummed up everywhere (and for just reason).

Every time a human gets in a car, they put someone else's life at risk.

The ultimate realization of self driving will be successful, and will have profoundly positive impacts on society

This is clearly possible in principle. After all, fully autonomous vehicles can have much more capable sensor systems than any human driver, can process the input from those sensors faster, and can then apply any necessary adjustments to the vehicle's behaviour almost instantaneously, all without ever getting tired or distracted. No human driver will ever match that potential.

The problem I have with this whole argument, and thus the reason I'm not sure the moral/ethical arguments stand up either, is that we don't know how to reliably write excellent software yet.

The entire argument for autonomous driving rests on the premise that we will at some point in the usefully near future reach a situation where the cars drive themselves much better than we humans collectively do. But there is, to the best of my knowledge, no evidence so far that we as a society can actually create the necessary software yet.

Until we can, any autonomous vehicle control system will be at risk of life-threatening bugs. In particular, it will be at risk of widespread failures of the same type that could cause much more harm than any single human driver's failure could, whether caused by defects in the software alone or by a malicious actor exploiting a security vulnerability. That is a huge risk that is getting far too little attention so far IMHO.

But humans too have life-threatening bugs, especially when combined with an automobile. Car+driver combinations have blind spots, for example. Human drivers get drunk, get distracted, and spill coffee in their laps. I think autonomous cars will kill people, and they will kill people in situations where human drivers would not. But I'm finding it increasingly easy to imagine that autonomous driving will save lives overall.

Yes, human driving is far from perfect and sometimes that has tragic consequences. But there are two issues at play here.

The first is whether we expect the autonomous vehicles to become safer than the population of human drivers they would be replacing under normal circumstances. This is a population measure, and it's about overall safety and averages, and given enough time and data it seems a realistic outcome.

The second is the risk of a single catastrophic failure, which is a risk that doesn't generally exist with today's human-controlled vehicles. It is no exaggeration to say that the kinds of autonomous vehicle technologies we are conceiving here have the potential, if they go wrong, to cause more harm than any WMD ever deployed in war. And not only the theoretical potential, but given the current state of our software development abilities, a non-trivial chance of it actually happening in practice.

Until we've dealt with the latter problem, which I don't believe we currently know how to solve, any arguments based on the long-term superiority of autonomous vehicles over the capabilities of any human driver is going to be fundamentally flawed.

> Personally, I also think we (humans) have become way too culturally risk adverse.

This is what it really comes down to.

Imagine if we invented the car today in our current cultural climate. I don’t think it would be possible for humanity to reap it’s advantages.

Back when it was invented people actually had to drive the things around on public roads that had a bunch of pedestrians walking haphazardly accross them continuously.

I was an autopilot skeptic, and it’s entirely for this reason. I didn’t think it would be possible for anyone to actually do what needed to be done to get this technology across the line. That is to say, give it to customers before it is safe.

If Tesla is actually able to keep doing this without it getting killed politically, then I think it actually will happen, and that gives me some hope for humanity more generally.

The car didn't replace walking, it replaced carriages, which were easily as heavy and could probably go much faster than the first cars. Sure, the horses would generally try to avoid obstacles, but horses pulling a heavy carriage at speed are not infinitely maneuverable. So, I doubt that cars would have ever seemed like a huge risk initially simply because it wasn't significantly different from what came immediately before it.

“The only way to push past those local optima is by acquiring large amount of real world automated driving data, using systems in production”

That, I don’t see ever being true. What would you get from that that you couldn’t get from “large amount of real world human driving data, using systems in production”?

If I understand your point (why not backseat systems?) then I'd say it fails to capture the complexity of feedback from the automated systems.

Or to put it another way, every time the driver's inputs diverge from the automated system's inputs, the test run is invalidated. You're no longer getting results of what the automated system did.

Try any system ever that has had a v2. Engineering is not omnipotent.

> Legal : If you sell it as a drive assist and not a driver replacement, you're off the hook, with a high probability

In Germany they lost a case about their Autopilot advertisement. So not even legally they are in the clear.

What was the case about? If it was in the opposite direction (that they branded a drive assist system as an autopilot) then it might actually help them now. "Look, we even have a court decision saying it's not an autopilot".

> ... using its customers as QA when they sell products that can kill people

They aren't just using their own customers, they are using all of us.

The cars won't be just crashing into inanimate objects, but school buses and ambulances. The QA is being done to all of us and our loved ones near their cars.

Tesla runs their new releases for millions of miles on the wide fleet in shadow mode before they release to limited beta. This is not an untested release in the way the article makes it seem.

That's not as useful as it might sound. If a particular action the driver didn't take would lead to results that didn't happen, then shadow mode will have a lot of trouble knowing that. In general it's a problem called "off policy evaluation" and it's really hard. Tesla has to estimate what would have happened counterfactually, had the bot been driving, instead of seeing what happens with the bot driving. A million miles of this shadow mode evaluation isn't worth nearly as many miles of actual self driving.

> If a particular action the driver didn't take would lead to results that didn't happen

I'm having a hard time parsing your double-negative, but when shadow mode is running, it's choices are back-tested in real-time with the driver's choices, so I'm not sure how this fails to replicate what a real-driver would have done in such scenarios.

Here an example. I’m on the freeway in my Tesla and instead of staying behind a car, at a safe following distance as predicted, I change lanes and speed past. Was it because, 1) the car was doing something I perceived as dangerous and I moved to a safer position, 2) I’m in a freaking Tesla and I can

Tesla's shadow mode is collecting real data up until the point where the car's decisionmaking process and the human driver disagree. Everything after that is simulated fantasy. And since human drivers aren't necessarily going to do "the right thing" in all circumstances (there isn't necessarily a single 'right thing') you can't treat 'car-disagrees-with-human' as an error case.

Other cars respond to your actions, so the moment your actions deviate from the bot the reactions of the other cars are no longer valid to test against.

Backtesting a driving model, it’s very hard to tell what would have happened, had the car deviated at all from what the driver did.

e.g. You see the driver swerve for no apparent reason, did the driver simply drive more recklessly than the car would, or did they avoid an obstacle the model didn’t catch?

And yet there are several recent videos of Tesla cars doing crazy stuff on the roads with this system.

Wait till you see the videos of humans doing crazy stuff on roads in cars!

If humans did the crazy stuff on the road at the same rate as we've seen from the few select hours of FSD driving uploaded by Tesla owners (who are fans and more likely trying to show it at it's best) this week, we'd all be dead.

Do people randomly brake at high speeds for no reason ? Tesla cars do:

https://m.youtube.com/watch?v=xj73Q3lWvFM

All the time. Witness the effects of cellphones/children/food/spilled food/makeup/jarring news/exciting music/significant others/attractive people/gawking at random bits of interest. Also, brake-checking just cause' and for insurance fraud.

If what you're proposing is that "success" in this experiment is that every Tesla now drives at least as well as the absolute worst-case human driver, then you've pretty much articulated why this experiment is so dangerous.

The thing is these are not the "absolute worst-case human driver", these are the absolute average human drivers.

People get easily distracted, get excited, get tired, all of which are propitious to bad decision-making.

Many human drivers are downright dangerous, but that's not a reason to add dangerous machine drivers on the road, either.

The adaptive cruise control in my Honda is also bad about braking without good cause. Of course, they don’t market it as “self-driving”.

Automatic emergency braking also has false positives.

Who is legally responsible for the car doing something crazy? Will engineers or executives go to jail?

How does that explain the persistent issue with phantom breaking ?

https://m.youtube.com/watch?v=xj73Q3lWvFM

It's been a problem for years now.

And it's scary as hell. Had it happen to me twice in a 600-mile highway roundtrip this summer. Was very happy nobody was following me closely, because the combination of regen breaks and disc breaks really stops the car quickly.

Still magnitudes safer than human drivers.

Drunk drivers and human mistakes are two things I accept in traffic (or rather - laws and human behavior both show we don’t accept them). It’s factored into my risk.

And FSD isn’t orders of magnitude safer. They are barely as good as human drivers and only in some conditions.

For the worst situations they aren’t comparable at all (since they won’t drive at all)

This must be repeated over and over for some reason: people will never accept self driving cars that are just as good or bad as human drivers. Nor should we. It’s a much too low bar.

Self driving cars will need to be orders of magnitude safer than human drivers to be even remotely acceptable.

If it's consistently as good as an average human that's still better than an average human. That's because robots don't get tired, hungover, angry, bored, drunk, or get an eyelash in their eye at a really bad time.

Yes. But again people won’t accept that. People will rather be killed more often by drivers that feel guilt, go to prison, have strokes or poor eyesight than less often by a machine that does not have all of those flaws but also none of the feelings and responsibilities.

I’d be thrilled if self-driving cars could meet that bar, because then I can feel at least as comfortable napping in the car as I do driving it. And unlike humans, I have the expectation that my car learns when other cars crash, so I have a lot of reason to expect that once the system rolls out it will continue to improve.

I hate almost everything about Tesla and their business model but the idea of full automatic driving, once vetted to even a bare minimum level of approximate human parity, is enough to make me consider one anyway.

Lets do a simple test:

You are in between two walls and there are two straight lanes. Lane 1 has Tesla FSD coming in at full speed. Lane 2 has a human driver coming in at full speed. You can choose to be in one lane. The FSD and the human can both see you only 50 mts out and if they brake as soon as they see you, you will survive.

Which lane will you choose?

Well, a Model S can just barely stop in 50 meters at 60mph, while some other luxury and sporty cars in the segment can stop in 35 meters or less. And that’s only 60mph, hardly “top speed” or even a normal highway speed. So regardless of driver assistance features, I’d choose a car that at least has a chance to even physically stop assuming instant reflexes. 50 meters is probably a much too small distance for this thought experiment.

I don't know my cars that well, but an alert driver will take at least 300ms to begin reacting. That's over 8m at 60mph (not sure why we're combining unit systems here). So I'll stand in front of the Model S in this particular scenario and hope to get away with 'mere' broken legs/pelvis. Give the human more time and is probably take the human, not because that's necessarily the more reliable option, but because its distribution of outcomes is more well known to me.

Most, if not essentially all, cars in that category have emergency automatic braking, so the Model S still isn't the best choice.

It absolutely isn't.

Hate to say it but I think you’re in the minority with that opinion. Reaction from most people has been “woah”.

Also, there is no moving fast and breaking things here in real terms. Drivers are and have to pay attention while driving, and Tesla is strictly monitoring this with beta. You also specifically have to opt-in to beta software. 99% of Tesla owners do not have this functionality.

I think we are mainly silenced by the hype folks who can't not tell the rest of the internet about their excitement. I'd like to think OP's opinion is the majority, hard to tell on a startup forum where most folks are hodling TSLA.

I would be astonished if a majority of people on HN have Tesla stock, for what it's worth.

Would you only be astonished if they directly hold it, or also if it's via a fund or three?

(I think you're probably right they don't, HN international and diverse enough, but I wouldn't be astonished.)

I meant "directly hold" (since I intended to reply to the parent's comment about "hodling").

I would guess that most people on here are cautiously optimistic. Given the progress that has been made over the past decade on all things AI my priors are telling me this will eventually work.

Note that it doesn’t mean I love it; I just expect that they will figure it out.

This is exactly how tech demos for all sorts of AI have gone for decades: initial impression is "wow, I can't believe it's doing so well!" But the positive impression quickly disipates as there's real world use of the product.

Tech demos have a way of setting good impressions that are not realistic. I have a feeling that's exactly what is going on right now, but a few more close calls with crashes and the shine wears off.

I'm not sure what you're looking at here. There's no "tech demo" going on here. It's a new beta of technology replacing an old and quite well functioning technology that is a SAE level 2 system. This is not frozen in stone. In fact Tesla is updating it every ~5 days right now. There's many people that post daily videos of how Tesla handles on a specific road near them and you can see it improve update to update.

Ironically, deep learning might finally be the solution to this, in a convoluted way.

Previous AI systems generally fell over when they hit the final 2% of real world scenarios, unable to cope with the explosion of complexity.

In contrast, modern AI finally provide a toolkit for "last mile AI" (proverbially speaking), by allowing incorporation of those scenarios without breaking the entire previous model.

All deep learning implementations to date fail on the last 2% of real world cases. Why would deep learning be the solution for its own shortcomings?

Failing on the last 2% (or whatever small number we want to set) isn't a shortcoming of deep learning: it's a shortcoming of all prior knowledge systems. Humans included.

And the barrier to surpassing that problem has been hardware (now approaching solving) and calendar- & person-time.

The problem with 1970s AI (ingnoring hardware) isn't that we couldn't build useful AI, but that we couldn't afford to build useful AI in most domains.

Calendar time costs money. People time costs money.

Deep learning and derivates finally provide a solution where we can substitute machine time and data volume for calendar and people time. As a result, we can afford to do things in domains we couldn't before.

> Failing on the last 2% (or whatever small number we want to set) isn't a shortcoming of deep learning

Citation needed. Deep learning is used to power some of the biggest money making machines in the world (like google or facebook ads) and they're still bad. It's also used to power products like voice recognition and it'll still add dog to my shopping list instead of soap.

I would counter that those systems are as good as they need to be. Google and Facebook have realized advertisers are dumb. So why waste a ton of people time optimizing past what moves the revenue needle?

Where I'd expect to see interesting developments (hopefully published openly) is in sentiment analysis / disinformation identification, as companies begin to see it as an existential threat to the unregulated nature of their businesses and fund accordingly.

Forget this 2% business. How does it do in snowy conditions at night? Because in some places, that's around 12.5%.

You cant go "woah" for 10 years. SD cars are not delivering. One more death and people will be calling to burn them.

One should do sentiment analysis on these discussions threads over the years to see how the attitudes have changed.

A much less dangerous version of this was the VR hype train

Tesla's unregulated and (some might say) reckless roll-out of self-driving technology could indeed be the biggest threat to the technology's realization.

As you point out, all it will take is a few fatalities for public opinion to harden against self-driving, and laws will start banning or severely restricting it. Rather than being wowed by Tesla's gung-ho approach, boosters of self-driving technology should be angered by it.

Totally disagree - the public is complacent about death insofar as it is not reported on by the media. If the media reported on daily accidents, and the carnage they bring, nobody would drive. In general, unless you assume this is going to be so bad that it can't be ignored, your prior should mostly be informed by how much you think the media will try to capitalize on a moral panic about self driving cars or not.

Wait. You disagree that the media will try to capitalize on a moral panic?

I phrased it poorly. I think the odds of the media turning self driving cars into a moral panic is quite low. I don’t think such a panic will materialize organically.

I don’t buy the FUD. By all measures driver assist programs make the road safer. These cars are not out on the streets massacring people like human drivers are. So meh.

What measure are you talking about here? Has Tesla published any stats about FSD?

The other stats in this thread make it sound like Tesla is not even the same league as other attempts at driving.

Yes, they have, lots of public data. The latest show that autopilot on is just slightly better than a human re. safety (but Teslas overall are about 10x safer than the average car already).

Tesla has not provided any numbers supporting that, and what numbers they have provided are so obviously flawed as to be either designed to mislead or produce by someone utterly ignorant of statistics. I’m assuming the former, but who knows.

But we are talking about FSD, not autopilot, a very different beast.

You’re asking for evidence that can’t possibly exist yet. They’ve only had the FSD beta out for a week, and only for a tiny subset of the fleet. It takes time to accumulate accurate stats and publish them. According to the NHTSA, there is one accident for every 479,000 miles driven. For Teslas with automatic safety features disabled (no AEB), the accident rate is 1/3rd the NHTSA average.[1] They’ll need tens or hundreds of millions of miles driven to get accurate safety figures.

1. https://www.tesla.com/VehicleSafetyReport

Many other FSD companies are doing testing with professional safety drivers, sometimes on closed courses or in specific cities with well-understood road systems. Presumably Teslas has done the same thing, and they have data from that? If they don't have such data, that means they're trying to develop this data using amateur Tesla owners on public streets all over the country, which seems quite a bit less safe.

Other manufacturers all rely on high resolution maps for anything approaching self driving. This is why Waymo is geolocked to a single city.

This is a beta release, you have to be quite gullible to think anyone would deploy untested software on the roads.

Thousands of people die each year because of stupid mistakes by human drivers. From a utilitarian perspective, pushing for self driving as fast as possible - in a manner that some might even judge to be reckless - is a net positive.

I'm surprised people have this opinion, on HN out of all places. In my view this technology shouldn't be allowed on public roads at all. Not even a little bit. Either it's 100% safe under all circumstances or it shouldn't be allowed. And Tesla definitely shouldn't be allowed to call it autopilot or "full self driving" when it very clearly isn't.

And yes, human drivers might be statistically worse - but just because this might be statistically better(and I'm not even sure that's actually the case here) it doesn't mean it should be allowed.

Isn't this position tautologically the case of letting the perfect getting in the way of the good?

I'm not saying Tesla is making a good move here or maybe isn't being careful enough. I don't know yet & am not familiar with how the engineers feel about the development of the product & how bad the corners they cut during development are. That being said, the standard you've laid out is not a good world to live in. I don't know if the tech equipped in the car is sufficient.

A huge statistical difference in quality should be sufficient to go with automated cars even if they don't make the same mistakes as humans or sometimes make mistakes humans don't. "Statistical difference" isn't about a coarse level analysis either but situational. If they get into fewer road accidents but kill pedestrians at a larger rate, that's a problem. The problem is the news media only reports the top-line number because analysis is hard.

Industrial automation that runs the world at large in thousands of ways isn't 100% safe under normal circumstances, much less "all" and yet we have almost entirely automated elevators, high voltage relays, medical equipment, nuclear reactors, autopiloted airplanes, and gas powered furnaces, to name just a few.

Most of this automation operates under certain assumptions about its environment and in some cases shuts itself down in a controlled fashion to fail safe in ways that may not be available to cars on roads, but these are not new ethical dilemmas.

But to pick just one example, automated saw stops and controllers aren't 100% effective at preventing injuries, but they're still a huge safety improvement over the human population at large just winging it.

Should everyone on HN have the same opinion, didn’t get that memo?

Nothing is 100% safe. By your logic we’d have no medicine, have you ever read the side effects of some medicines.

> By your logic we’d have no medicine, have you ever read the side effects of some medicines.

You picked the worst example possible.

Medicine is one of the most tightly regulated fields in the world with huge upfront cost and approval processes spanning years or even decades. E.g. the research and approval process of a new antibiotic costs multiple billion dollars before it can be sold.

What you see now wrt the Covid-19 vaccine is an international effort that is fast-tracking everything. Regulators are willing to overlook a lot for it to come to market as fast as possible and it's still taking a year or more before it's approved for use on everybody.

I would actually say it's a very good example. That no matter how tightly and strenuously we regulate the safety of a product, 100% safety is an unattainable goal and that given sufficient benefit there is an acceptable level of risk of grave outcomes, including death. This is from a publication on the risk/benefit evaluation of new medical devices from the FDA:

Uncertainty – there is never 100% certainty when determining reasonable assurance of safety and effectiveness of a device. However, the degree of certainty of the benefits and risks of a device is a factor we consider when making benefit-risk determinations.

pg 11 - https://www.fda.gov/media/99769/download

Regulated doesn't mean safe. It means a little, but not a lot, compared to the internal practices of the companies under regulation.

And regulation in no way - in no way - addresses the fact that medicine isn't close to 100% safe before adoption, and neither should self-driving cars be. As long as they are safer than humans, that's enough. Any other conclusion will cost lives.

If technology is required to be 100% safe under all circumstances, then why aren't humans held to the same standard? For humans, we have a driving test that measures readiness for driving. It includes a margin of error for new drivers and there's also an expectation new drivers suck more than experienced ones (e.g., indirectly via higher insurance rates).

I'm having a hard time understanding your position as to why the technology should not be allowed, even if it were to perform statistically better than humans. Is this more of a philosophical perspective?

What else not 100% safe shouldn't be allowed? Passenger planes? Cars in general? Cancer medicine?

Why wouldn't a community of engineers welcome a statistical improvement over meat drivers?

If that's the standard, full self-driving won't become a thing for another 50 years at minimum if not longer, because not only is it a crazy high target, the lack of data and improvements brought by real-world usage would hamstring development — there's not but so far test tracks and single-city pilots can go.

It's a lot like the "old space" approach to spacecraft engineering, spending decades and hundreds of billions of dollars to develop a rocket that would've been cutting edge half a century ago, all in pursuit of making the initial product perfect.

>Either it's 100% safe under all circumstances or it shouldn't be allowed.

This would be a much more compelling argument if the technology in question weren't replacing an existing system that is frequently lethal. I do agree that it needs more testing and possibly regulation.

I'm surprised that people have this opinion, especially on HN out of all places. Seems like most people here are aware of how dangerous human drivers are and the vast possibilities of technology. Seems like you're just needlessly scared since you're arguing for killing people?

> Either it's 100% safe under all circumstances or it shouldn't be allowed. And Tesla definitely shouldn't be allowed to call it autopilot or "full self driving" when it very clearly isn't.

Then no human should be driving on the roads either or at least every vehicle should have a breathalyzer in it to check your blood alcohol level before the engine can turn on. You're being unreasonable.

I’m trying to think of a comparison and I can’t come up with one. Something where we’ve relinquished control to something else that we’ve designed. Maybe trusting robots to build the cars themselves or something akin to that.

Either way, it’s definitely new territory for us. I am sure there will be debates on this forever but I foresee reasonable arguments by both sides.

Elevators. Automatic doors. Fly-by-wire. Airplane autopilot.

Heck, modern cars are full of things that control aspects of driving for you: Cruise control. Automatic emergency braking. Brake assist. Antilock brakes. Traction control. Many of these features are so beneficial to safety that they are mandatory for new cars.

If the goal is to prevent traffic fatalities then funding self-driving research must be one of the least effective methods. Tens of billions of dollars have been spent and no lives have been saved.

If we could achieve cycling usage at the level of the Netherlands, mass transit usage on the level of Japan, and road safety on the level of Norway then we would cut road deaths by 90%. These are ambitious goals but can actually happen and are not an open-ended research project.

How do you know no lives have been saved? I've seen footage of Teslas autobraking to avoid collisions ahead that happen a couple of seconds later. Does that not count?

Well the government isn't exactly funding self-driving very much. Indeed if the government actually had such goals then funding cycling paths, mass transit routes would certainly be better, but the government doesn't have such goals really. So companies develop technology that fits within the government funded systems that exist (namely the government heavily funds building highways and roads) so the systems must be built for that application. Feel free to encourage politicians to change their ways, but until that happens automated vehicle research will continue.

Cycling usage and mass transit are strongly related to the culture of society. I believe funding self-driving is more cost-effective than getting Americans to get rid of their cars and use mass transit (and I say this as a non-American that doesn't have a car, and uses mass transit, bike and walking extensively).

>These are ambitious goals but can actually happen and are not an open-ended research project.

can actually happen *if you live in an urban area

Cool, would you volunteer to be one of the extra casualties in order to make self driving work? If not, then why volunteer others?

Has anyone actually been killed by any Tesla vehicle that was driving automatically (that wasn't in the car itself)? Even on the earlier SAE level 2 "Autopilot" system? I don't remember hearing of any personally.

I don't recall any incidents where a third party was killed due to autopilot, however, I don't believe it's reasonable to extrapolate the general case performance based on prior special case performance, especially when the prior special case performance did in fact result in deaths that would clearly have been avoided by a human driver.

Indeed, but we won't have general case performance until it is acquired. And the statistics show that the prior special case performance did reduce accidents compared to without using the system even if some accidents happened that would not have otherwise happened.

This argument is silly since it assumes that self driving is somehow both the fastest and most efficient way to save people's lives, however we have working technologies we could implement today that could save lives (breathalyzers for cars, measures of visibility plus warnings, etc). Self driving efforts related to saving lives miss the opportunity costs of doing so.

There's no reason both paths couldn't be pursued.

That's only true once the technology seems to be better than human driving. That doesn't seem to have happened so far, and is definitely not true of Tesla's efforts.

By the same logic, in march we should just have started distributing whatever attempts at covid vaccines companies thought might be promising. Sure, it might kill a lot of people, but think how many others could be saved! It's reckless to require demonstrated safety and efficacy of drugs!

What's missing, using this analogy, is that it is only a net loss to slow down the vaccine that actually works, which we don't know a priori. But if you fast track something that kills a bunch of people, then it might have been a waste. You might not learn enough for it to be worth the cost.

What if we push out Tesla's cars and all we learn is that vision based neural networks with the particular models tesla's using today aren't good enough? How many lives would you sacrifice to learn that when a big investment in a more careful testing strategy might say the same thing?

To say nothing of the criticisms of the utilitarian perspective. If we believed in it as a society, we'd round up the poor/unskilled/uneducated for mandatory medical experimentation, but thankfully that idea is horrifying.

To be a safe driver you have to be able to predict what the people around you are going to do. Simply reacting is not enough. I don't think that it's a coincidence that the guy selling self-driving cars is the same guy who goes around exaggerating the current abilities of AI.

I too am shocked that anyone is ok with this, given the auto industry's history of fighting against safety and transparency.

We humans are terrible at predicting the behavior of other humans, we have unconscious biases and our own behavioral patterns that regularly cause us to misinterpret data and make the wrong decision. Quite honestly an objective AI that also has faster reaction times is likely to be a better driver than any human, it’s just a matter of training data.

Note: I’m not going to say Tesla isn’t being at least moderately reckless with its rollout of self-driving features, just that I firmly believe even a less than ideal AI with good enough training and sensor data can trounce the safety record of human drivers.

> AI with good enough training

This is the crux of the issue. If we had AI that could outperform human drivers and make our roads safer then I'd agree that we should use it. But I see no indication that this is the case, or that it's even technologically possible at this point.

Tesla can make whatever claims they want, but until their self driving systems have been independently and rigorously tested they shouldn't be anywhere near public roads. Have such tests happened yet? The article doesn't mention any, so I assume that they haven't.

This is exactly why regulations such as professional engineering licenses were put into place. You don’t want someone with that mentality building a bridge.

No, this is exactly why vehicle regulations were put into place.

Regulate and validate the vehicles and their software updates, not the engineers.

Edit: Or get used to our field moving as slowly as the bridge building field.

Huh? Bridges stopped failing when engineers were held responsible. That’s a pretty fast change.

Holding people responsible doesn’t decrease or slow down innovation. It alters the incentives to align the innovation with public benefit. If properly caring for society is too slow for you, your part of the problem.

> Bridges stopped failing when engineers were held responsible.

Oh did they?

https://en.wikipedia.org/wiki/List_of_bridge_failures#2000%E...

Edit:

> Holding people responsible doesn’t decrease or slow down innovation.

Citation required. This is strongly non-intuitive.

> Oh did they?

First one only sagged, no catastrophic failure, no deaths, no injuries.

A few more on that list were over a hundred years old. Not sure when he started holding engineers responsible.

Then you have the bridges rammed by ships, terrorism, construction accidents, a wooden suspension bridge in the middle of nowhere that had an upper limit of three and was used by seventy, etc. .

Seems as if the bridges themselves have held up fairly well.

Perhaps a less conservative bridge development culture would have resulted in more innovative bridges that could have avoided those incidents?

History shows that less conservative bridge culture will reduce costs at the detriment of safety.

Engineering education is designed to ensure you know better than risking lives for fun or profit.

It doesn't prevent you from trying to introduce new technology as long as risk is acceptable. Otherwise we'd still ride horse carriages (with special horses bred to be slow).

> I really don't understand how a company can get away with using its customers as QA when they sell products that can kill people.

Stuff like that is only possible in the US. Would be unthinkable in the EU. Maybe in UJK but you know, they are no longer ...

In the long term this will cause EU auto makers to get their self driving tech from US companies (either through licensing or partnerships). VW has already partnered with Ford. Most of the other european auto makers are getting their tech from Aptiv. When it comes to self driving, none of the EU car manufacturers can compete with Waymo, GM (Cruise), or Tesla. Even when they manage to build really cool stuff, it gets shut down by regulators.[1]

1. https://europe.autonews.com/automakers/audi-bmw-others-frust...

Maybe EU auto makers just wait, meaning work on something, until they can put something on the market that works. After watching some hours of youtube footage I can say one thing for sure: Teslas' FSD does not.

By that time the US companies will have technology with a track record of real world use. The EU companies will have an easier time buying that tech than building their own.

This is analogous to banning airplanes until they can be proven safe to fly in. The countries that don’t ban them will lose more lives early on, but they’ll have a technological advantage. The place where this analogy breaks down is that hundreds of people are killed by human driven cars every day. Every year cars kill 120,000 people in the US & Europe. If we can develop and deploy this technology a year sooner, we can save tens of thousands of lives.

I think only the Tesla approach wouldn't fly in Europe, but trained drivers should hold, last year there was some announcement by Renault and Waymo for something in Paris though hard to tell what it actually meant: https://techcrunch.com/2019/10/11/waymo-and-renault-to-explo...

Tesla Autopilot is already used all over the EU.

The underlying motivations explain all: this isn't about improving automobile safety anymore (if it ever was): it's about recognizing deferred revenue from this feature (which has been sold for years) and trying to keep up, from a marketing perspective, with Waymo and Cruise. That's why they're not using trained safety drivers, but fanbois.

Waymo is at least as safe as the average human driver, while Tesla has unleashed literally worse-than-drunk drivers upon us all. Heaven help us.

The ethics of this are heavily influenced by how much you believe a) this rollout will result in a net decrease in automobile accidents and b) how much we should assume drivers will both be informed and intervene if the system missteps. If you believe both are unlikely, or you believe engineers should have a "do no harm" based ethics similar to medicine, this would be unacceptable. But if you believe both are highly and/or think engineering ethics should focus on harm minimization, this makes sense to do and under certain assumptions it becomes unethical to not roll it out.

As a concrete example: if you assume that 99% of accidents that would be caused by the system will be prevented by driver interventions, and you think overall the system will reduce the likelihood of an accident by 20%, the question you have to ask yourself is: if 1 in 5 people harmed over the next 6 months we could have saved, should we avoid doing so and wait until only 1 in 200 will not be harmed, in part due to their own negligence? As you can see, the assumed probabilities matter a lot, so I'd be curious how one can come up with good projections. The only case where these probabilities don't matter is if you believe that engineers ought to never create harm where none would have existed otherwise, but when it comes to self driving cars, this is an impossibility since it assumes perfect autonomy.

In general, if you buy into the theory that "human + AI" is always going to be smarter than just AI, then arguably even in a world with great autonomous driving, the best system will be one that has both an AI and a human. And that is what we have here, so it's more a question of if AI quality is sufficient or if this is just a bad idea in general given the natural tendencies of humans to be poor co-pilots.

> I really don't understand how a company can get away with using its customers as QA when they sell products that can kill people.

They already claim that drivers must pay attention to the road in order to prevent accidents and this has been the status-quo in the automotive world for a century or so. Selling products that can kill if misused is not remarkable.

This could turn ugly if it's shown that enabling self-driving results in much more accidents but there is no data to support this.

> get away with [...] products that can kill people

if your moral premise is saving lives, your conclusion does not follow. vehicles are a leading cause of death and injury, so this omelette is worth breaking a few eggs, no?

AV has the potential to save a lot of lives. of course, there will be effects in other markets like insurance, new car sales, road repair, junk/salvage yards, tow trucks, etc. what the net economic effect will be is up for debate.

False dichotomy fallacy. AVs are great. There's a way to test them that doesn't involve killing people: you put a safety driver in the car, make sure that they're attentive at all times, and record what the car's software does vs. what the driver has to do. Then you gradually ramp up to more sophisticated scenarios as the software gets better.

Waymo and Cruise do it this way, and their software gets better without killing anyone. Uber and Tesla figured they'd play it fast and loose and killed people.

You're assuming eggs need to be broken in the first place, which is not at all proven. There are more options than what Tesla is doing - just look at Waymo as a counterexample of actually prioritizing safety over public release speed.

If you want the amount of data necessary you do need miles driven. The millions of extra miles may very well be what it takes to put tesla over the edge. Sure you can do that in a controlled environment, but it would take a lot longer, and people would die in the mean time.

This argument makes sense, if you can assume that Tesla will actually deliver functional FSD system as a direct result of their approach. There's no data to support it.

You're claiming "even though Tesla's self driving might be better/safer than human driving today, and will in aggregate save lives over the present state, there's this other approach which could save even more lives!"

It's not certain that Waymo's approach will be better, especially at scale (and in cities other than Phoenix). We don't have that kind of data yet. And even if it is better, Tesla's approach is still a pareto improvement over now. The perfect is the enemy of the good.

Shortcuts don't necessarily get you there sooner either. Tesla's approach might be risk without the reward. It's not a deaths-vs-FSD-date dial you can just turn (not even a dial you have to turn).

Well if it is worth that eggs get broken then it should at least be Tesla paying for it no? They seem to want to have it all, advertise their product as autopilot (not drive assist), use their customers and everyone else as Guinea pigs and take the data to improve their systems, but if things happen it's the drivers fault for not being attentive and they have to take financial (and moral) responsibility.

If they really thought some eggs need to be broken to get _their_(because they are using the data nobody else) systems up to speed, they could just come forward and say it and more importantly pay for the consequences.

AV has that potential in principle, sure. It hasn't realized it yet, particularly in Tesla's implementation.

> they sell products that can kill people

All car manufacturers sell products that can kill people.

> All car manufacturers sell products that can kill people.

Yes, that's exactly why most of them have much better QA than Tesla.

The key difference is that only Tesla ships one that will autonomously kill other people.

Except tons of other manufacturers have SAE level 2 systems that autonomously follow road curves and will fully run into objects that are unexpected. GM Super Cruise will kill people all the same.

Super Cruise is geofenced to known highways. What "road curves" are you talking about?

Presumably the curves in roads.

Well, I own a Tesla Model X when I'm with my family I don't engage self driving car mode, nobody force bus to do it. And when I do it is normal situations like 101 Highway

>nobody force bus to do it.

And where do the other people on the road factor into the decisions of you and your 5000+ lb vehicle?

Just watch out for left-hand exits like the HOV lanes at the Hwy 85 split going southbound.

Let's start "Falsehoods self-driving programmers believe about roads:"

All highway exits are to the right from the right-hand lane. If the left lane line curves away, the road is curving and you should follow it.

The only stats I have seen on this show that the autopilot reduces the rate of collisions. So it's not really a product that kills people, but the opposite. The stats came from Tesla, though, so I can't rule out the possibility that they are incorrect.

Tesla stats are extremely misleading and deliberately slanted toward favoring Tesla. Unfortunately, it's not really possible to find real stats since they're hidden by individual OEMs and insurance companies. But you would need to compare comparably priced modern cars with standard ADAS features to Teslas with Autopilot to get a datapoint worth talking about for the general case. You would also need to look at where Autopilot is engaged (mostly highways where accidents are rare in general) vs. not engaged (mostly city streets where accidents are common) to draw any real conclusions about safety.

> Tesla stats are extremely misleading and deliberately slanted toward favoring Tesla. Unfortunately, it's not really possible to find real stats since they're hidden by individual OEMs and insurance companies.

You realize that those two statements are in conflict with each other right? If they're misleading and favoring Tesla then you need contrary stats to state that. However you state that they're not available, so your previous statement can only be a lie.

Sorry, that’s not a logical conclusion. Just because real stats are hard to come by does not mean that Tesla’s stats are true.

But until you have evidence that they aren't true, what reason do you have to believe they aren't true? All you're really arguing here is that you personally believe Tesla is a "bad" company so you should disbelieve anything they say. You do realize that's not at all convincing and just shows you're personally biased, right?

But it does mean that you don't know that your first statement is true.

In Tesla's most recent report, it turns out that compared to Teslas using plain-old-human drivers, Teslas using Autopilot driver assist functions were 2x more likely to get into an accident, and Teslas using FSD were 3x more likely to get into an accident.

Any way you cut it, Autopilot and FSD are statistically worse than human drivers.

This is the only data Tesla has released comparing similar cohorts of vehicles (i.e., modern luxury cars). The only stats in which Tesla beats the average driver is where they include all cars ever put on the road, including semi trucks...but under those models every other modern car is also better than average.

[EDIT: And this is the Forbes article on Tesla's report. There's also a Cleantechnica post on the same report as well. https://www.forbes.com/sites/bradtempleton/2020/10/28/new-te...]

I’m pretty sure you misread the article. City driving is 3x more likely to result in an an accident compared to highways, not self driving. The numbers are one accident every 1.7M miles for autopilot in city conditions vs an accident every 500k miles global average (including highways, for which Teslas ratio is 1 in 5 million miles).

No, I read the report. You've inverted the numbers.

If Tesla FSD was 3x safer than human driving Musk would be bragging about it. The fact that he's not indicates that my interpretation is the correct one.

Again, I don’t know what you’re looking at. Who’s claiming it’s 3x safer now? Where are you getting these “interpretations”? Just read the article carefully.

> [...] made me think twice about buying a 25+ year product that keeps my home powered and dry.

does it even become profitable before the 25 year mark in the current market?

This comment strikes me as completely disingenuous.

(1) None of the companies involved have ever had the motto "move fast & break things." Zuckerberg created that philosophy to Facebook. It's foolish to apply it an entire industry.

(2) They are no more "using its customers as QA" than pharmaceutical companies do when gradually ramping up trials that also kill people. There has been heavy testing before this point, and it seems entirely reasonable to scale up testing before a general release.

and that's why it's not allowed in the EU

You have to continue to leave your hands on the wheel and pay attention.

Just like having a trained bear chauffeur you around. Except people see that as a bad idea.

It’s not a public beta. The select few drivers know what they are getting into.

It’s strange that people would rather a small chance of crashing themselves, versus a smaller chance of the car crashing for you.

If Tesla is going to be a safer driver than me, it’s already a good investment. If a software bug means I die, well, human error might too.

I'm less concerned about the Tesla driver dying (since it means they weren't attentively monitoring their car right?) and more concerned about the driver they t-bone or hit head on due to a software bug.

They're selling safety statistics. I find that unethical. You don't say to a widow "but think about how many times I could have killed someone and didn't".

Also note the double speak of Musk on the topic.. he was the first to talk (even boast) about how the structural design made the car the safest of all .. but now safety is not as essential, it's ok to have some accidents. And it will improve their machine learning data set right ! /s

That would be an argument worth having if a tesla with FSD were actually safer than a comparable modern luxury car driven by someone in the same demographics. Or even just compared to a tesla without FSD. We aren't even at that argument yet.

That's only the case if this new Autopilot iteration is actually a safer driver. There are many who doubt that.

It clearly isn’t. Everybody should ask themselves, “How often do I have a near-miss on the road that another human in the car with me prevented by warning me?”

I’ve had that happen a few times in the decades I’ve been driving. It appears to be a regular occurrence in this software.

Cars driven by humans kill people at a higher rate than this shitty AI, why doesn't anyone care then?

I don't understand this point of view. Cars WILL kill people with AI or humans driving. AI is not great right now, but it's already way better than humans in several situations.

Then people get spooked by bugs that the AI brings because they're not "normal" accidents, and think it's a bad idea to continue to train the AIs.

But why? Why is a "beta" AI killing people worse than the normal everyday drunk/distracted/bored/texting human driver?

> AI is not great right now, but it's already way better than humans in several situations.

Every stat that I've seen that comes out with the machines better than the people isn't a like-to-like comparison. Every like-to-like comparison I've seen has the machines still worse than people, so we don't even need to have that argument. Beware the propaganda.

Every stat? Wow. I'd be very interested in reading some of this material if you could link to some.

If you take a shitty driver and compare it to an AI, I doubt the AI will perform worse when it comes to preventing accidents.

Sure, a skilled human driver will likely perform better than this early stage AI, but if even 1% of the bad drivers out there switch to an AI, that's a net positive in my view. Not to mention all the other risky situations involving alcohol, sleep deprivation, etc. where an automated response can save lives.

You didn't address my main point though. Why is a driverless accident worse than a drunk driver?

I don't believe you can claim that humans kill more people than shitty AI since we haven't seen a world in which shitty AI is driving around unassisted. Given that Uber/Waymo/Cruise all have safety drivers working, it's hard to say how many fatalities would have been caused if those safety drivers weren't there since that would require predicting the future at the moment the safety driver disengages the AI.

By “cars kill people” do you mean car defects or human drivers? If former do you have a source for this claim?

I meant cars driven by humans. Edited right after I posted.

Reply on news.ycombinator.com