Felony charges are first in a fatal crash involving Autopilot | Hacker News Reader

Felony charges are first in a fatal crash involving Autopilot

apnews.com

57 pointstiahura142 comments

Fully self driving cars are far further off in the future than most in the sector would publicly admit. In controlled environments there’s a lot of potential today and things like self driving trams at airports are nothing new. But self driving in the “real world” mixed with humans and all the crazy one-off things that happen in such scenarios is not something that will happen at scale anytime soon.

There are still far too many edge cases to work out and for all the amazing things that “AI and ML” can do, handling edge cases is something this technology is actually quite terrible at relative to humans.

I'll go a step further and say they aren't just further off but impossible to achieve until someone can reasonably prove otherwise. We simply aren't going to see cars without a steering wheel and manual controls (i.e. Level 5 automation) for general purpose use on public streets in our lifetime.

Yep. I would argue that driving is AI complete - in the sense that only true Artificial General Intelligence can do it, and so do everything else to.

Perhaps this is because it requires cooperating with other General Intelligences (i.e. us) in an open world.

I'm still mystified as to why no country has switched over to ultra-light rail. The costs of asphalt projects seem to dwarf the cost of doing anything reasonable and leave AI as an exercise for the over confident.

https://www.youtube.com/watch?v=tBJ0GvsQeak

can't you get a ride in phoenix like today with waymo? or is not open to the public?

I mean I understand that their AI is not perfect but it doesn't seem like a pipe dream

Average human is actually quite terrible in driving cars, as evidenced by the global death toll numbering in dosens of thousands every year. Even halving this number would be a win, losing a lot of people to such edge cases. However, the nature of the problem would not allow for it, because every death from a self-driving car will get excessive coverage and public outrage, and people will prefer losing far more people to human errors than fewer to computer errors.

The annual US death toll alone is dozens of thousands; the global number is well over a million.

I don’t think that’s the case- I don’t think there’s any evidence that self driving cars are better drivers than human drivers in real world conditions.

If this were only a human psychology problem, someone would just dump a fleet of self driving cars on the road and prove they have the tech. The rewards are too good to ignore and surpass any potential fine/regulation, if the tech is good enough.

But it’s not there yet, it might never get there. We’re so far off general AI it’s unreal, e.g. a human can drive a car with a single eye open, and one hand on the wheel, and another on a throttle- all from the drivers seat. Even with sensors and data coming in from all over the car, AI can’t manage it.

I didn't take the parent to be saying that self-driving is better, just that humans are really surprisingly bad, and that's the bar we should be measuring against, not an ideal world where no one dies in traffic accidents.

The main ethical issue of 'deaths due to self-driving cars' is because of who being at fault?

If the car is self-driving, and truly self driving - no steering wheel, when it causes an accident, who is at fault?

Humans make mistakes, but humans also can change their response easily to a given input. Is Tesla at fault? Do we just issue them a fine for a person's life? Should the car do its best to preserve the life inside the vehicle, or the lives outside the vehicle? Automobiles do significant damage to pedestrians in regards to their dangerous design.

Surely corporations should be held to a higher standard than people, as they're not bound by the same restrictions a single person is.

I don't see how that's any different of an issue for when any machine malfunctions and kills someone. Was the design or manufacture of the machine negligent in some way? Was it a failure mode that we've decided is rare enough that no one is at fault?

Self-driving cars may be new, but deaths caused by machines are as old as machines.

Machines in a business setting when they cause death, often it is due to improper safety protocol for said machines.

Consumer devices don't have this level of control over a person's life anywhere near as often, in the sense that the device has your life at stake and makes decisions for you.

"no one is at fault" is entirely counter intuitive to engineering principles. When a rare accident happens, the engineer is often responsible.

We don't 'build' AI in anywhere remotely the same way as buildings or legacy software products, there isn't a 'designer' for said AI system other than the training model - which can certainly not account for every possible edge case.

Self-driving cars can't 'only be \better\ than humans'. Because only humans can currently drive. Self-driving cars need to be nearly 100% safe when you entirely remove the human driver.

When machines malfunction and kill people, the manufacturers pay. If it was gross negligence that killed someone, someone goes to jail, unless they're the dead one.

Every day, our homes don't catch fire despite all of the electricity running through the walls, our combustion engines don't explode, our batteries don't explode, our buildings don't collapse. The modern world is full of things that could kill us and don't.

I didn't say no one is at fault, the manufacture is at fault if they built the vehicle in a way that we deem was wrong enough to deserve fault.

And we get to decide who's liable, just because you can act like the AI is sentient doesn't mean you can't make the company who wrote the code responsible for the actions of the AI.

And if we wait until self-driving cars are 100% safe, hundreds of thousands of preventable deaths will happen in the meanwhile.

Come back and make this comment when we actually have something better than human driving.

Human walking, human biking.

This is a very underappreciated point. SO many people die from driving each year, but a lot of people treat a single death from self-driving cars as a reason to scrap the whole endeavor.

Put into scale, there were 1.11 deaths per 100,000,000 miles driven in the United States in 2019. The average human is actually surprisingly good at driving cars, we just drive 3.9 trillion miles per year so the absolute numbers of fatalities are enormous.

I say this as a person who genuinely feels like people are generally terrible at driving.

Should we improve these numbers? Of course. But AI-driven cars will have to beat a pretty high floor of capability here. It's entirely possible they're already there but it's difficult to know for sure given the biased way the numbers are calculated. The manufacturers claim that their systems are much safer drivers than humans, but those systems also hand control back to the human driver during difficult or unpredictable situations. It's easy for the statistics to swing in AI's favor when they only have to count the "easy" majority of miles while shifting blame back to humans for the hard parts.

Humans are quite good at driving cars, as evidenced by the death toll per million km. In fair comparisons self-driving cars aren't even close.

Yes people would prefer to have agency in their lives and die based on their own decisions instead of a nameless programmer (a large fraction of the victims have themselves to blame for their death, this would change with self-driving cars).

> In fair comparisons self-driving cars aren't even close.

Please reference a fair comparison.

AFAICT, we only have the unfair comparisons from Tesla, which say that Autopilot has an accident rate 1/5th compared to when it's disabled, and that Full Self Driving Beta has 0 accidents total. Neither of those comparisons are fair, but that's the best we have, AFAICT.

Isn't that a bit of a false dichotomy? GP is specifically calling out that self driving technology has limitations (and specifically as they pertain to car accidents being edge cases in the first place). Self driving tech hasn't proven itself to be better than human drivers outside of controlled environments, nor are these two options the only options available (e.g. more investment in public transit and alternate forms of transport come to mind as being very closely related to the problem space)

By all means, improve the tech and show us that it can in fact put out better-than-status-quo safety numbers, but it's important to not confuse potential with wishful thinking.

Humans are actually really good at driving - we had just assumed it was easy because anyone could do it.

Driving plays to our two key strengths - tool use, and cooperation. Indeed, the car becomes an extension of our body, and the road is our forum.

Don't forget all the sensory processing stuff that comes with being a predator that eats other mobile animals.

The main barrier is the state of the tech, not media outrage.

There is a gap between the current state of the tech, and computers being better at driving than humans, that may be impossible to bridge. In order to bridge it, as a society we would need to accept a period where these things start to become more common, but they make our roads less safe.

Only through trial-and-error in blood can this technology improve. I don't think we will collectively see it as a worthwhile experiment.

Must there be blood? How many people have died because of Waymo driving? How many because of Tesla or Uber?

All the more reason for Tesla to avoid sleazy sales tactics like calling adaptive cruise control "self driving", and claiming that cars will be able to drive themselves coast to coast before 2020.

People _might_ overreact to deaths from true self driving cars, but we don't know that yet because such vehicles don't exist.

Full self-driving cars limited to highways is probably a much more reasonable goal. A lot of the issues seem to be with intersections and pedestrian crossings - things that are absent on highways. Plus, infrastructure to assist self-driving cars like beacons on off ramps could feasibly be added.

"infrastructure to assist self-driving cars"

That would seem to be a good investment for the industry as a whole, yet the trade group doesn't seem very visible. Possibly the sort of thing that should have been in BBB.

Small, route-constrained mass transit vehicles would solve 80% of this with 20% of the cost. We could call them trollies or buses.

And 10% of the convenience, comfort and utility of having your own car.

Cities already have buses. Rural areas don’t want them.

I think that's the key. If self-driving cars ever reach a level of ubiquity, it's going to be on large, open roads and highways. Urban areas will still require a human driving.

Why don't we have an Uber for buses yet? Has this been tried anywhere? Not necessarily a for-profit company, but an app that people could use that would offer potentially better routes / trip times. So the bus wouldn't have to hit locations where no-one was getting picked up / dropped off, etc..

It has been tried in a few places. Here's one: https://www.cbc.ca/news/business/uber-lyft-ride-hailing-on-d...

I'm using these a couple of times a month: https://www.moia.io/en

> Rural areas don’t want them.

Da'fuq? Rural areas love and use the heck out of buses - school buses.

As for me, I'd love to have a bus which came within 3 miles of my "rural" neighborhood. Save on gas, insurance, and be able to read when I have to commute to the office?

I can't remember the last time I used a school bus to get to the grocery store or park

I know. It's strange how they have such great coverage for kids, but decide that it's not worth it for adults with adult money.

I wonder how good “public transport” could be if it matched the money inflows of private cars. Possibly amazing.

Think customisable bus routes, cheap subsidised taxis, a fast bus every small-n minutes and so on.

Clearly we won’t find out as it’s a massive sunk cost, but I think this alternative reality could be great.

Having lived in places with both amazing and terrible public transport, I understand the skeptics, but also know just how amazing public transport can be.

In London the cost of public transport is comparable to running a car. The end result is meh. It works ok but with the usual drawbacks.

Yes, I wouldn't put London on the "great" pile. Adequate. Tube is OK for longer distance, but the friction of getting on and off is a pain. Buses have to crawl through heavy traffic and pedestrians everywhere.

Warsaw in Poland IMHO has very good public transport. Reliable, timely, used to be cheap when I lived there (not sure how much it went up). Never struggled to get from A to B.

Idk what prices you are paying.

It varies but a zone 1/2 annual travel pass is £1,444 presently.

Excellent vehicles for facilitating a pandemic - packing many passengers from different social circles together in an enclosed space.

Not if the air is filtered.

https://jamanetwork.com/journals/jama/article-abstract/27714...

> Fully self driving cars are far further off in the future than most in the sector would publicly admit.

Full self driving have NOT been solved for highways in a desert.

This is literally the easiest real world scenario imaginable. Highly and strictly regulated environment, and no obscurities like snow happening. I don't think there is a hope we will get any close to self drive with current technology.

Is the sudden object in front of me a block of concrete that fell of or stray page from newspaper floating by on a wind gust? Hard bake or ignore it?

We cant possibly capture all edge cases of real life. You need systems that have cognitive capabilities to deal with real life

Or optionally different type of transportation that is not so complex to navigate and understand. There are already fully automated metro lines. Maybe solution is to change the problem itself.

It is expert defence against charges like this that will undo the whole nearly-but-not-quite autonomous driving industry. And rightly so. I am stunned the industry thinks effectively an [OK] click through is going to save them here.

(And before anyone quotes million of miles per fatality stats, this is a legal question on a single case, generalities don't cleanly apply.

Edit: I think Tesla and co. will need to show that it is reasonable to assume a driver can quickly notice and respond to a hazard particularly after being lulled into passivity for what could be hours by the drive system. Good luck.)

What are some analogous legal outcomes where companies got users to agree to take liability for the company’s product outcomes and it failed in court?

In aviation, I’m pretty sure that so long as you can disable an autopilot, anything that happens is the human’s fault. Guns - nothing to do with the makers and everything to do with the shooter.

I can’t think of any examples where it worked.

Most guns aren't self aiming and self firing.

Well, in aviation, they do go to great lengths to let the pilot know which mode they're flying in and transition into and out of that mode has it's own protocol, training and feedback. Indeed the whole surepticious engagement of the MCAS system on the 787-MAX aircraft, and in particular not including updates for pilots in training to deal with what to do when that system fails -- i.e. awareness -- is probably going to end a few careers at Boeing. That seems to be an example very close to the Tesla situation. The AF447 crash in 2009 has a spookily similar pathology -- confusion about automated flight control modes. Both incidents were awareness of automation issues, one with it being on, one with it being off (more or less).

Tesla is banking on a beep and/or input control shake covering them for mode change awareness for a completely novice, untrained and uncertified driver. Is that a reasonably assumption?

Edit: There were other (tragic) factors in the AF447 crash, but in that case, if they'd re-enabled autopilot and let go of the controls, they would have very likely survived.

Post-edit-window-edit: I retract "re-enabled autopilot" - . There was whole procedure required if there were residual airspeed discrepencies on all three sensors. e.g. Autopilot would only help if at least two were non-iced. AF447 may have had zero.

https://www.cbc.ca/news/world/airbus-gives-new-warning-on-sp...

http://www.aero-news.net/index.cfm?do=main.textpost&id=e7851...

I don't know if it is purposeful or the name is genuinely confusing, but Autopilot is basically adaptive cruise control and lane assist. Updates that happened after this crash took place always stopping at stop signs and red lights for properly equipped cars.

FSD or Full Self Driving is the thing that drives the car all by itself. If you are not in the beta, it can only be activated on highways.

So basically this guy plowed a red light and cruise control and killed someone.

Some agency really needs to enforce consistent naming for all these features. Vague marketing terms like "autopilot" and "self-driving" cause enough confusion to be outright dangerous. A car manufacturer can't advertise ABS, airbags, tire pressure monitoring, traction control etc. and then say oh we define these terms differently. Autopilot should be no different.

Is Tesla's definition of autopilot way different than the definition used in aircraft?

Does the ordinary consumer have any idea what an aircraft autopilot actually does?

Yes.

An autopilot in a commercial passenger aircraft is fully capable of controlling the plane after takeoff, landing, and taxiing up to the destination gate. Or in other words, after takeoff, it is fully capable of controlling the plane without any input from the pilot.

So, when Tesla advertises their cruise control as "autopilot" they are overstating the abilities of autopilot mode if they put in the fine print that a human actually is responsible for controlling the vehicle and that autopilot is just a glorified cruise control.

Aircraft autopilots are glorified lane-keeping and assistance devices that steer a course in an environment that is reserved for airplanes, with clear lines of sight, clear slots for the individual vehicles and substantial separation between all vehicles in the lane. The environment is free of pedestrians, cyclists, children, footballs and other participants in traffic. If Tesla wants to subscribe to the definition of “autopilot” with the capabilities profile used in aircraft, it must only be usable in comparative environments - that is: a few hundred feet above ground, for takeoff and landing with instrument assist and with pre-assigned slots and fixed routes.

Great explanation. Thanks.

We don't have pilots in cars, so yes, it is very different.

I don't see the relevance? Seatbelts, brakes, steering, engine all mean something completely different when on a passenger jet vs a sedan.

Yes, it is materially different, and they tackle wildly different problems in different fields, with again wildly different sensing and control interfaces, different uncertainties and challenges, and unbelievably disparate levels of required training, certification, and competence for use.

The desire to name it "autopilot" stems from a desire for positive marketing, not a desire for technical correctness. (And certainly not from a desire to accurately convey limitations to users!)

But if it was named something different the the stock would crash. :-/

I'm a bit conflicted on this. If you actually examine what "autopilot" does on aircraft, it doesn't seem misleading at all. Most autopilot, unless it's connected to a terrain database, will happily fly you straight into a mountain. Autopilot usually just maintains heading, altitude, and airspeed.

It's difficult to compare flight autopilot and automobile autopilot.

Flight autopilot generally warns when terrain or traffic is advancing. You can also program routes, etc. on the flight management computer which connects and controls autopilot. Many planes even autoland in certain conditions.

Do consumers know this?

Agreed.

Growing up, I spent a lot of time on boats. The county I grew up in was primarily a fishing/crabbing economy. Autopilot on boats is like this. It maintains a compass heading. At no point does it steer you around obstacles or react to anything other than changes in the compass heading induced by drift, waves, currents, wind, etc.

When I read the headline, I thought the operators of the FSD tech were being charged: Tesla or Uber. To literally slam the brakes on this, hold the companies accountable rather than individuals who believed that the products would work as advertised.

If the supposed FSD still requires an individual to fully man the vehicle, what is the point of having it? It seems like bloatware surveilling you with little benefit to the user similar to what modern phones do.

To give your car a devil tempting you to do something stupid like let down your guard and trust it to do the right thing which it will usually do right up to the point it doesn't?

Exactly, what is the point? It’s also clearly misnamed.

Level 5 or bust.

Unfortunately, I think that holding the driver responsible is more legally feasible route to take. If you are successful with that, consumer confidence in these features will drop to 0 and the companies will be forced to put up or shut up with their advertising claims.

That’s ridiculous. Ruin multiple lives for a product that failed.

But now that I say that. Was it Auto-pilot which legal is cruise control? Or was it Full-Self Driving and the car made a decision? Legally it’s a matter of man vs machine of it is FSD. Is man wrong if there was even the slightest attempt to correct the machine’s decision after the fact? If it is in fact legal to allow said machine to act while supervised.

It could also be a matter of arguing current training and regulation for FSD, is regulation of drivers sufficient for what now is a new mode of private transportation? Should autonomous vehicles be allowed to be owned yet, do they really fit under the same umbrella as normal license? Or should they be special like a CDL or MJ licenses?

> When I read the headline, I thought the operators of the FSD tech were being charged: Tesla or Uber.

It would be harder to charge them with criminal violations, but civil charges are possible against them (and the manufacturer) even if the driver is criminally culpable.

FSD is not involved in this case.

Elon and his sales team should be charged. They are selling this FUD. You can't sell something and then blame the user that they should have read the fine print.

Are you sure you meant to use “FUD” in this sentence? It doesn’t make sense. “Elon and his sales team (“they”) are selling this fear uncertainty and doubt”.

The result of "autopilot" is entirely predictable. The NTSB, NHTSA, CSB, and pretty much every other safety board have repeatedly found that any system that requires someone (anyone) pay complete attention, while also not actually doing anything, will fail in the second part. It is wholly unreasonable to expect a person "in control" of a partially self driving car to have instantaneous response, it isn't even reasonable to expect 90% attention to what is going on around the car.

Add to that Tesla's insistence on intentionally mislabeling things - autopilot has a clear implication, especially given their demos of it, the "full self driving" mode is clearly outright false labeling - and I think Tesla does share liability.

The current model of "self driving" with the expectation the a driver will be able to immediately respond to anything the driving system misses, is inherently unsafe.

There are countless HCI studies showing the problem that you mentioned - any interface that requires full and immediate context 1/10000 times while being hands off otherwise will fail in real world use.

This issue is so pronounced in mission critical systems that the problem is intentionally inverted, where the machine intervenes in the 1/10k scenarios and humans are in control the rest of the time - think flight control laws in aero or auto scram in nuclear engineering etc.

There was no "fully self driving" in play with this accident. The driver has the responsibility to control the car. He let the car drive through a red light. He's wholly at fault.

(for background: I've driven tens of thousands of miles using Autopilot in both a Model S and Model X, on both local roads and highways)

As long at Tesla continues to willfully and fraudulently push a product called "Full Self Driving" -- onto a customer base they know to be largely naive and unsophisticated -- they share responsibility for whatever death and carnage inevitably results from this cynical and manipulative course of action.

I'm always fascinated by this urge to push individual lapses and judgment onto large and wealthy entities.

I own a Tesla Model 3 and I have chosen not to purchase the full self-driving package nor do I intend to rent it. I don't think the technology is worth it at this point to me personally.

When you own the vehicle, it is very clear that the autopilot mode is simply a lane assist system. They have messages flashing all over the screen stating that it will not stop at stop signs or red lights. It's not like it's hidden from anybody and instead is blatantly displayed with no way to stop it from displaying every time it is activated.

This driver was abusing a system that is meant to reduce fatigue. He should bear the consequences. Those of us who enjoy the reduction fatigue when driving from Denver to Omaha like I've had to do recently, should not have the feature disabled because of his irresponsibility.

Do you think the product name "Full Self Driving" has anything to do with reality?

Tesla bears some responsibility, but it's also on an individual to sniff out bullshit when necessary, and not be gullible to the point of committing negligent homicide.

When you operate heavy machinery, it's your responsibility to read the owner's manual, and not just listen to the marketing.

"Full Self Driving" means "Part Self Driving"? What a country!

He wasn't using Full Self Driving, nor was he a customer.

Autopilot != FSD. Autopilot comes with the vehicle. Full Self Driving is an add-on. He was on Autopilot.

But let's not the facts or details stop us.

You may answer the question, if you wish.

The customer base are rich and can take care of themselves. Everyone else on the road is who need to be protected.

The product being used was not "Full Self Driving". He was using "Autopilot" which comes with the vehicle. FSD is a completely separate, and expensive, upgrade.

Thoroughly enjoying the completely uninformed comments on this.

"Autopilot" is still highly misleading.

Maybe to you. “Autopilot” is a term I don’t think has an incredibly clear meaning. Planes are known to have autopilot, but I think of it like “we’ve reached cruising altitude, going to point this bird at LGA and let the autopilot keep us on the right route”. I don’t know what the analogy would be for cars. Maybe something like adaptive cruise control plus lane keeping and auto braking?

Then call it "adaptive cruise control" and be done with it.

There's only one reason they pick the term "autopilot" -- and that is to give people the impression that these cars have magical powers that they do not, in fact have. And until proven otherwise, most likely will never have.

Unfortunately, the entire thread is people who don't understand that Autopilot and Full Self Driving are not the same thing. Autopilot comes with the vehicle, and is simply a lane-assist system. It explicitly tells you it will not brake the vehicle when activated. This driver was using Autopilot.

Full Self Driving is an expensive upgrade, and the driver wasn't using it.

But not to worry, the conflation of the two is driving dozens of confident, self-assured statements in this thread that are embarrassingly off-the-mark.

Have you both proven that the so called 'driver monitoring systems' used for Autopilot and FSD not only doesn't work, but it neither properly checks that the driver behind the wheel is paying attention with their eyes on the road?

Perhaps that why these days both of these systems are open to abuse constantly by the driver without the car monitoring them and hence the cause of these crashes regardless if it is autopilot or FSD.

I have autopilot, fsd and now fsd beta. i’ve read all the comments up till now. most of the opinions seem to come from a good place but not experience. before i say “ ask me anything” let me start out by saying, within 20 seconds of activating any of these features, you need to shake the wheel to prove you’re there. there are plenty of clues before that that indicate this is not a hands off affair, but this alone is where you know it’s not going to be a ‘set it and forget it’ . ok, ask me anything.

Does Tesla Autopilot normally stop at traffic signals?

If not, seems pretty cut and dry - the driver is at fault.

If Autopilot does stop at signals, then it's a bit more muddied to me.

When the accident happened, it didn't. That was added in 2020.

Didn’t Gen 1 “autopilot” had this? Mobileye has had traffic light and even speed limit sign detection for its cruise control for years..

It does not, "autopilot" is what Tesla calls adaptive cruise control and lanekeeping assist. Attempting to obey stop signs and traffic lights is part of "full self driving."

https://www.tesla.com/support/autopilot

Only defense I can imagine is that he was misled by Tesla's "autopilot" naming and and thought the car would drive itself.

Outside of that the story is basically "driver puts on cruise control, blows through red light without braking, kills two people."

EDIT: since the crash was in 2019, here's an older version of that page http://web.archive.org/web/20190906125836/https://www.tesla....

> Autopilot is an advanced driver assistance system that enhances safety and convenience behind the wheel. When used properly, Autopilot reduces your overall workload as a driver. 8 external cameras, a radar, 12 ultrasonic sensors and a powerful onboard computer provide an additional layer of safety to guide you on your journey. Cars built between September 2014 and October 2016 include one camera and less-powerful radar and ultrasonic sensors.

> Autopilot is intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any time. While Autopilot is designed to become more capable over time, in its current form, it is not a self-driving system. There are five levels of automation and Autopilot is currently classified as a Level 2 automated system according to SAE J3016, which is endorsed by the National Highway Traffic Safety Administration.

I can't say that 100% for certain since Tesla's features come and go with software updates, and the above excerpt doesn't get very specific on capabilities. But I don't believe it knew how to stop for stoplights at the time.

That's what I thought.

I still hate Tesla's marketing. Autopilot is anything but. FSD isn't. Etc. I wish FCC/FTC/somebody (not sure which agency could regulate that) would smack them for it.

That doesn't sound like something that would reduce my workload as a driver. Sounds like I need to be with hands on the wheels, always on the lookout for the system not to respond to something that it otherwise should, AND for things that it shouldn't. Waiting for the system to respond and acting when it doesn't means my response delay has actually increased, making me more susceptible to accidents. Am I wrong?

That seems somewhat selective quoting of the page to me. The video on the same page prominently starts off with "The person in the driver's seat is only there for legal reasons. He is not doing anything. The car is driving itself".

So Tesla wants to have it both ways, to boast of self driving above the fold, while denying the self driving in the small print.

> The misuse of Autopilot, which can control steering, speed and braking, has occurred on numerous occasions

It feels wrong to call it misuse when you’re literally using it the way Tesla has previously, is currently, and will continue to market it.

Was there a title edit or something? Why are people talking about FSD? This seems to be about autopilot

On Tesla, commenters don't read before commenting.

will be fascinating to follow this one and see how responsibility for the death is apportioned.

I can see the argument for indicting the driver. He could be seen as negligent for choosing to unleash this car, known to be buggy, on a public street, leading to two deaths. His counterargument is obvious too, that the manufacturer assured him of its safety, and its therefore the manufacturer's fault.

Which then begs the question - if it is Tesla's fault, does that mean manslaughter charges will be brought against a corporation? What would that even mean?

IANAL, but corporations have been successfully charged and convicted of felonies in the US:

https://en.m.wikipedia.org/wiki/List_of_companies_convicted_...

At the time, Autopilot did not stop at red lights. It was (and mostly still is) a lane-following cruise control. How could it be Tesla's fault?

Tesla does not promote FSD as being autonomous, the driver is going to have a hard time blaming Tesla:

https://www.tesla.com/support/full-self-driving-subscription...

Note: These features are designed to become more capable over time; however the currently enabled features do not make the vehicle autonomous. The currently enabled features require a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment.

if the car automatically “suddenly and unintentionally accelerated to an excessive, unsafe and uncontrollable speed", even an attentive driver could find themselves unable to prevent the crash. it strikes me as more akin to a car with faulty brakes. such a car would not be certified as fit for the road. perhaps self driving should not be either.

Calling it “full self-driving” is promoting it as being autonomous, regardless of what the disclaimers say. If it’s not capable of that, they should call it something else.

Right? You can't have a label say X, and then in small print say that it absolutely is not X. It's equivalent to a product saying "Contains absolutely no nuts!" and then in small print saying "may contain nuts".

Full Self Driving means that the car is capable of driving itself, not that it can handle all situations autonomously.

A counter example, the package of Cheezits says "made with 100% real cheese", even though they are not 100% cheese, cheese is the 3rd ingredient, below flour and oil, but of the cheese used in the product, it's 100% real cheese. (so it makes it a meaningless statement, I guess they mean "no artificial cheese flavoring")

That's not a very good argument: "made with" implies that it is one of several ingredients. If Cheezits said "made entirely of" then the consumer would be understandably confused if there are other ingredients. Think about what you would answer to the questions "Did you make this pizza WITH sourdough?" vs "Is this cake made entirely OF lego?".

If the car can't "fully" self drive, then it shouldn't be marketed as such. Call it "partially self driving", "assisted self driving" or whatever.

That depends on your definition of driving, for example, take the dictionary definition:

"To guide, control, or direct (a vehicle)."

My car has driver assistance features, it can do basic lane keeping, keep a set distance behind the car in front (even in stop and go traffic), and has emergency braking assistance. But no one would say that it's self-driving.

Tesla FSD can control the car from entering the highway to taking your exit (plus some support for driving on smaller streets).

But it's not autonomous, hands-free driving.

I am not sure I understand the argument here as what you said highlights that Tesla's FSD is not fully self driving.

Taking the dictionary definition of fully:

"completely or entirely; to the fullest extent"

I would expect a fully self-driving car to drive for the entirety of any trip I wish to take. Anything less than that I would consider as "partial".

The car can fully self drive -- it has full control over steering and brakes, can pass other cars and even enter/exit the highway.

My car has limited self-driving, it can only do limited lane keeping or keeping distance from the car in front.

So you're defining it as the capability to automate steering, accelerator and brake? (i.e. the three things that constitute driving)

By that definition self-parking cars can also fully self-drive.

My definition of driving is being able to do all the tasks required to pass a driving test, i.e. driving around a city (intersections, roundabouts etc.), motorway, country road and car parks.

That’s clearly a bogus argument.

Made with 100% real cheese

Is clearly and unambiguously not the same as

100% cheese

But it is quite a good example:

A cheesit contains an ingredient that is 100% cheese.

A Tesla with FSD mode, is a car that has a feature that is full self driving.

If that mode cannot handle all conditions, then it is not full self driving. That means no “I’m confused” switch to a human driver.

Now explain the meaning of "Full".

The disclaimer is clear, the FSD means that the car is capable of full self-driving, but is not capable of autonomous driving without a driver paying attention.

Given that Tesla hasn't been sued into oblivion by drivers who got into accidents by incorrectly treating FSD as autonomous driving, I think the argument will hold up in court.

I’m not trying to be obtuse but I genuinely don’t understand what the difference between the terms ”full self-driving” and “autonomous driving” are. They’re synonymous to me, and I have to imagine they are to the average person.

Tesla explains it as:

Autopilot is an advanced driver assistance system that enhances safety and convenience behind the wheel. When used properly, Autopilot reduces your overall workload as a driver. Each new Tesla vehicle is equipped with 8 external cameras, 12 ultrasonic sensors and a powerful onboard computer provide an additional layer of safety to guide you on your journey. Model 3 and Model Y built for the North American market have transitioned to camera-based Tesla Vision, which are not equipped with radar and instead rely on Tesla’s advanced suite of cameras and neural net processing to deliver Autopilot and related features. Model S and Model X continue to be equipped with radar.

Autopilot comes standard on every new Tesla. For owners who took delivery of their cars without Autopilot, there are two packages available for purchase, depending on when your car was built: Autopilot and Full Self-Driving Capability.

Autopilot and Full Self-Driving Capability are intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment. While these features are designed to become more capable over time, the currently enabled features do not make the vehicle autonomous.

That's bullshit. Name it something accurate. Don't give it a name that is an outright lie and then correct yourself in the fine print.

That sounds harder than just driving

The driver wasn't using FSD, so that's irrelevant. He was using Autopilot, a much less capable system that doesn't cost an extra $12k.

I think both Tesla and the driver need to be held responsible separately, through legal means.

It's the FTC's (or some other gov agency) responsibility to enforce that marketing claims make sense. They should go after Tesla.

At the same time, it's on a consumer to control their vehicle. It's on them to know this. The government should help consumers not get into legal trouble by enforcing accurate marketing claims, but if it shirks its responsibility there, consumers do not get a free pass.

So, manslaughter for the driver, fraud for Tesla.

What is the bug that lead to the deaths, and what assurances of safety were given by the manufacturer?

I'm probably in the minority in the HN audience that thinks this... but we are a long way off from fully self driving cars that can be used anywhere across the US. Be it Waymo, Tesla, or Cruise the problem is that these systems have way too many edge cases. For example, what happens when the sun is "blinding" you and prevents identification of the traffic light? What happens when there's construction and a construction worker gives confusing instructions to follow? This isn't unique to Tesla-- I've seen waymo cars nearly force a cyclist into a car door in broad daylight.

In Tesla's case they've also been particularly aggressive with releasing self-driving tech to the end user without much stipulation on how safe it really is. I wonder how many more easily preventable deaths will have to follow before people crack down on such lax rollout of autonomous vehicle tech.

I don't think the self-driving technology is good enough, agreed. But the legal and cultural practices they develop to fit these into already existing frameworks and expectations is also technology.

I think they're likely to have enough expertise at these other skills to enable a national rollout of self-driving cars even if they aren't safe enough. And after all, we have a lot of practice ignoring road deaths and a lot of legal precedent placing that burden anywhere but on the car manufacturers.

I don't think a handful of preventable deaths will create any real obstacles. Like I said we already have a lot of those. Plus as usual the people most likely to be hurt are going to be non-car users in pedestrian-hostile places, which are also harms we don't really prioritize addressing.

I also think, and someone is very likely to bring it up here, the "WELL BY THAT LOGIC" argument does a lot of rhetorical favors for the self driving cars. Any argument you can make about self-driving cars being dangerous is also likely an argument about cars being dangerous. Which is true, but if you acknowledge it you get discredited as a universal car hater.

Anyway there are a lot of good reasons not to allow automated cars on public roads right now. Unfortunately I don't think any of those arguments will stop them.

> What happens when there's construction and a construction worker gives confusing instructions to follow?

Indeed and I believe this is one of the primary failings of the nascent self-driving car industry; driving is seen as an engineering challenge, rather than an engineering and social challenge. When viewed solely as an engineering challenge, the task is daunting enough, but maybe tractable if you squint and target only a subset of driving conditions. But add in the social component of driving, which is largely being ignored by this new industry, and it's really impossible with today's technology.

Another example I saw on HN before: What happens when the tech is good enough to 100% avoid sudden pedestrian collisions, so pedestrians wise up and realize they can just jaywalk in front of any self driving car with impunity?

Especially in dense areas like NYC, you can solve the safety problem, but that may require these cars to drive too passively to ever get anywhere.

Perhaps that will the push needed to finally separate pedestrian and vehicle traffic. Even in NYC, pedestrians are largely treated as meat sponges who are going to lose the fight with a vehicle. It's much worse in other, less walkable cities.

When you put the shoe on the other foot, and give the power to the pedestrian, drivers will absolutely push to get more places to drive where pedestrians aren't allowed.

I think you get a lot of people trying to categorize the number of "years" before we get a given technology, but it's probably more accurate to measure the number of "unsolved problems", and my totally unqualified opinion is that there are still a lot of unsolved problems left before we get level 5 AV.

I don’t think you are it’s clearly the case. I also think MobilEyes approach is far better than Tesla’s and they might actually win the race even if they are hardly competing in it openly.

I do like the idea behind their safety model and the approach that the difference between L3+/4 and L5 is about reducing the MTBF to essentially 0 rather than just adding more compute and features.

They also are probably the only company that A) has had multiple generations of their hardware and software in a lot of different types of vehicles on the road for over a decade and B) don’t undersell the problem and oversell their solutions at all.

As far as I’m concerned they seem to be the only responsible player in the market right now.

In my opinion, "AI" is still far closer to being able to recognize a picture of a cat than it is able to cope with all the complexities of driving that a person might encounter and deal with quite easily.

What might be a lot closer is a restricted kind of driving that might cover 90% of people's usual cases. Restricted set of roads (well mapped and maintained), maybe specific instructions for problem sections, markings and/or signals that the car understands, maybe local updates for changed road conditions, that kind of thing.

We still have to train Google to recognize images of busses with captchas.

I think the suit against Tesla will be the more interesting of the two but I guess there’s no news on that yet.

It definitely has the potential. This quote is in the article:

> Lopez’s family, in court documents, alleges that the car “suddenly and unintentionally accelerated to an excessive, unsafe and uncontrollable speed.” Nieves-Lopez’s family further asserts that Riad was an unsafe driver, with multiple moving infractions on his record, and couldn’t handle the high-performance Tesla.

Are they suggesting that Tesla should have known and therefore should not have allowed the vehicle to be on autopilot or to drive at such high speeds? That's a HUGE claim. Tesla collects a lot of data, and they analyze the data. I don't think that this is disputed, but have they now accepted liability for allowing the car to operate at full capability with the knowledge that he's a bad driver? That would be a major blow, and would reverberate throughout the entire industry.

You are very right - this could be really interesting.

Whatever happens it could end up being the major precedent over assisted services/self driving; depending on how it’s structured.

"I hate this car. It is so complicated and it doesn't even take gas!"

Not clear is why it took so long for charges? Crash in Dec 29, 2019, charges in October 2021, almost two years...is this Tesla holding up data?

I think it’s the prosecution gathering evidence and deciding on whether to charge and what with. Justice system is slow

There is nothing uncommon about this. There are many jurisdictions in the USA where charges for most crimes other than very serious stuff like murder are usually raised only just before the statute of limitations runs out.

The "speedy" part of the sixth amendment is interpreted very loosely.

The "speedy" part was to prevent the government from locking people up indefinitely while awaiting a trial that was never going to happen.

Reply on news.ycombinator.com