Drivers react to Tesla’s full self-driving beta release
arstechnica.com
arstechnica.com
I really don't understand how a company can get away with using its customers as QA when they sell products that can kill people. It's one thing if it's a website for throwing virtual sheep at another; nobody really cares if your Facebook feed omits an update or two. It's another thing if a software bug means that you die. Or at least it should be - I'm amazed at the number of people who seemingly don't care.
I also wonder what this is doing to Tesla's brand reputation. I was just in the market for solar panels and Tesla's offering (both the panels and the solar roof) was very, very attractive on price & aesthetics, but their reliability record with their cars made me think twice about buying a 25+ year product that keeps my home powered and dry.
Also, there is no moving fast and breaking things here in real terms. Drivers are and have to pay attention while driving, and Tesla is strictly monitoring this with beta. You also specifically have to opt-in to beta software. 99% of Tesla owners do not have this functionality.
(I think you're probably right they don't, HN international and diverse enough, but I wouldn't be astonished.)
Note that it doesn’t mean I love it; I just expect that they will figure it out.
Tech demos have a way of setting good impressions that are not realistic. I have a feeling that's exactly what is going on right now, but a few more close calls with crashes and the shine wears off.
Previous AI systems generally fell over when they hit the final 2% of real world scenarios, unable to cope with the explosion of complexity.
In contrast, modern AI finally provide a toolkit for "last mile AI" (proverbially speaking), by allowing incorporation of those scenarios without breaking the entire previous model.
And the barrier to surpassing that problem has been hardware (now approaching solving) and calendar- & person-time.
The problem with 1970s AI (ingnoring hardware) isn't that we couldn't build useful AI, but that we couldn't afford to build useful AI in most domains.
Calendar time costs money. People time costs money.
Deep learning and derivates finally provide a solution where we can substitute machine time and data volume for calendar and people time. As a result, we can afford to do things in domains we couldn't before.
Citation needed. Deep learning is used to power some of the biggest money making machines in the world (like google or facebook ads) and they're still bad. It's also used to power products like voice recognition and it'll still add dog to my shopping list instead of soap.
Where I'd expect to see interesting developments (hopefully published openly) is in sentiment analysis / disinformation identification, as companies begin to see it as an existential threat to the unregulated nature of their businesses and fund accordingly.
One should do sentiment analysis on these discussions threads over the years to see how the attitudes have changed.
A much less dangerous version of this was the VR hype train
As you point out, all it will take is a few fatalities for public opinion to harden against self-driving, and laws will start banning or severely restricting it. Rather than being wowed by Tesla's gung-ho approach, boosters of self-driving technology should be angered by it.
The other stats in this thread make it sound like Tesla is not even the same league as other attempts at driving.
Regulate and validate the vehicles and their software updates, not the engineers.
Edit: Or get used to our field moving as slowly as the bridge building field.
Holding people responsible doesn’t decrease or slow down innovation. It alters the incentives to align the innovation with public benefit. If properly caring for society is too slow for you, your part of the problem.
Oh did they?
https://en.wikipedia.org/wiki/List_of_bridge_failures#2000%E...
Edit:
> Holding people responsible doesn’t decrease or slow down innovation.
Citation required. This is strongly non-intuitive.
First one only sagged, no catastrophic failure, no deaths, no injuries.
A few more on that list were over a hundred years old. Not sure when he started holding engineers responsible.
Then you have the bridges rammed by ships, terrorism, construction accidents, a wooden suspension bridge in the middle of nowhere that had an upper limit of three and was used by seventy, etc. .
Seems as if the bridges themselves have held up fairly well.
It doesn't prevent you from trying to introduce new technology as long as risk is acceptable. Otherwise we'd still ride horse carriages (with special horses bred to be slow).
If Tesla is going to be a safer driver than me, it’s already a good investment. If a software bug means I die, well, human error might too.
I’ve had that happen a few times in the decades I’ve been driving. It appears to be a regular occurrence in this software.
Also note the double speak of Musk on the topic.. he was the first to talk (even boast) about how the structural design made the car the safest of all .. but now safety is not as essential, it's ok to have some accidents. And it will improve their machine learning data set right ! /s
And yes, human drivers might be statistically worse - but just because this might be statistically better(and I'm not even sure that's actually the case here) it doesn't mean it should be allowed.
Nothing is 100% safe. By your logic we’d have no medicine, have you ever read the side effects of some medicines.
You picked the worst example possible.
Medicine is one of the most tightly regulated fields in the world with huge upfront cost and approval processes spanning years or even decades. E.g. the research and approval process of a new antibiotic costs multiple billion dollars before it can be sold.
What you see now wrt the Covid-19 vaccine is an international effort that is fast-tracking everything. Regulators are willing to overlook a lot for it to come to market as fast as possible and it's still taking a year or more before it's approved for use on everybody.
Uncertainty – there is never 100% certainty when determining reasonable assurance of safety and effectiveness of a device. However, the degree of certainty of the benefits and risks of a device is a factor we consider when making benefit-risk determinations.
And regulation in no way - in no way - addresses the fact that medicine isn't close to 100% safe before adoption, and neither should self-driving cars be. As long as they are safer than humans, that's enough. Any other conclusion will cost lives.
Most of this automation operates under certain assumptions about its environment and in some cases shuts itself down in a controlled fashion to fail safe in ways that may not be available to cars on roads, but these are not new ethical dilemmas.
But to pick just one example, automated saw stops and controllers aren't 100% effective at preventing injuries, but they're still a huge safety improvement over the human population at large just winging it.
Why wouldn't a community of engineers welcome a statistical improvement over meat drivers?
Either way, it’s definitely new territory for us. I am sure there will be debates on this forever but I foresee reasonable arguments by both sides.
Heck, modern cars are full of things that control aspects of driving for you: Cruise control. Automatic emergency braking. Brake assist. Antilock brakes. Traction control. Many of these features are so beneficial to safety that they are mandatory for new cars.
I'm having a hard time understanding your position as to why the technology should not be allowed, even if it were to perform statistically better than humans. Is this more of a philosophical perspective?
I'm not saying Tesla is making a good move here or maybe isn't being careful enough. I don't know yet & am not familiar with how the engineers feel about the development of the product & how bad the corners they cut during development are. That being said, the standard you've laid out is not a good world to live in. I don't know if the tech equipped in the car is sufficient.
A huge statistical difference in quality should be sufficient to go with automated cars even if they don't make the same mistakes as humans or sometimes make mistakes humans don't. "Statistical difference" isn't about a coarse level analysis either but situational. If they get into fewer road accidents but kill pedestrians at a larger rate, that's a problem. The problem is the news media only reports the top-line number because analysis is hard.
It's a lot like the "old space" approach to spacecraft engineering, spending decades and hundreds of billions of dollars to develop a rocket that would've been cutting edge half a century ago, all in pursuit of making the initial product perfect.
This would be a much more compelling argument if the technology in question weren't replacing an existing system that is frequently lethal. I do agree that it needs more testing and possibly regulation.
Then no human should be driving on the roads either or at least every vehicle should have a breathalyzer in it to check your blood alcohol level before the engine can turn on. You're being unreasonable.
What's missing, using this analogy, is that it is only a net loss to slow down the vaccine that actually works, which we don't know a priori. But if you fast track something that kills a bunch of people, then it might have been a waste. You might not learn enough for it to be worth the cost.
What if we push out Tesla's cars and all we learn is that vision based neural networks with the particular models tesla's using today aren't good enough? How many lives would you sacrifice to learn that when a big investment in a more careful testing strategy might say the same thing?
To say nothing of the criticisms of the utilitarian perspective. If we believed in it as a society, we'd round up the poor/unskilled/uneducated for mandatory medical experimentation, but thankfully that idea is horrifying.
If we could achieve cycling usage at the level of the Netherlands, mass transit usage on the level of Japan, and road safety on the level of Norway then we would cut road deaths by 90%. These are ambitious goals but can actually happen and are not an open-ended research project.
can actually happen *if you live in an urban area
if your moral premise is saving lives, your conclusion does not follow. vehicles are a leading cause of death and injury, so this omelette is worth breaking a few eggs, no?
AV has the potential to save a lot of lives. of course, there will be effects in other markets like insurance, new car sales, road repair, junk/salvage yards, tow trucks, etc. what the net economic effect will be is up for debate.
It's not certain that Waymo's approach will be better, especially at scale (and in cities other than Phoenix). We don't have that kind of data yet. And even if it is better, Tesla's approach is still a pareto improvement over now. The perfect is the enemy of the good.
Waymo and Cruise do it this way, and their software gets better without killing anyone. Uber and Tesla figured they'd play it fast and loose and killed people.
If they really thought some eggs need to be broken to get _their_(because they are using the data nobody else) systems up to speed, they could just come forward and say it and more importantly pay for the consequences.
I too am shocked that anyone is ok with this, given the auto industry's history of fighting against safety and transparency.
Note: I’m not going to say Tesla isn’t being at least moderately reckless with its rollout of self-driving features, just that I firmly believe even a less than ideal AI with good enough training and sensor data can trounce the safety record of human drivers.
This is the crux of the issue. If we had AI that could outperform human drivers and make our roads safer then I'd agree that we should use it. But I see no indication that this is the case, or that it's even technologically possible at this point.
Tesla can make whatever claims they want, but until their self driving systems have been independently and rigorously tested they shouldn't be anywhere near public roads. Have such tests happened yet? The article doesn't mention any, so I assume that they haven't.
Legal : If you sell it as a drive assist and not a driver replacement, you're off the hook, with a high probability. It's on the drivers if they get complacent. There may even be precedents with similar products for trucks.
Moral: pushing this now will very likely kill some people that would otherwise be alive (drivers or pedestrians). But it's very unlikely to be a carnage - I'd dare say even impossible. On the other hand, moving the technology forward and directly into widespread commercial use will save people, and likely even... what's the equivalent word for a "carnage" but where people are saved? Miracle.
Branding: well, there you have a real risk. Public opinion is fickle, and a particularly gruesome accident might do some real damage, both to Tesla and to self driving technologies in general. But then, they do have a high risk high reward mentality, and the chance is good that the first bad accident (statistically inevitable) will come after the advantages are proven and well established.
In Germany they lost a case about their Autopilot advertisement. So not even legally they are in the clear.
This is the only rationale I can come up with for Tesla's approach. The tenets go like this:
1) The ultimate realization of self driving will be successful, and will have profoundly positive impacts on society (e.g. safety, urban planning, ease and availability of transportation, economic)
2) The path to that ultimate realization contains many local optima traps
3) The only way to push past those local optima is by acquiring large amount of real world automated driving data, using systems in production
From a technical standpoint, I'm disinclined to disagree with them on the necessity of large amounts of real world system behavior data as a key ingredient.
Personally, I also think we (humans) have become way too culturally risk adverse. Or, to put the calculus another way: if I have a credible belief that I can save some number of future humans, what is the acceptable moral number of current humans to place at risk and/or kill?
Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."
From a purely ethical standpoint (leaving legal aside), the crux would seem to be informed users. Not all Tesla drivers are likely capable of understanding the current SotA, and I'd hope (and expect) that Tesla's phased rollout of the beta is informed by previous user interaction with assisted driving.
That, I don’t see ever being true. What would you get from that that you couldn’t get from “large amount of real world human driving data, using systems in production”?
Or to put it another way, every time the driver's inputs diverge from the automated system's inputs, the test run is invalidated. You're no longer getting results of what the automated system did.
This is what it really comes down to.
Imagine if we invented the car today in our current cultural climate. I don’t think it would be possible for humanity to reap it’s advantages.
Back when it was invented people actually had to drive the things around on public roads that had a bunch of pedestrians walking haphazardly accross them continuously.
I was an autopilot skeptic, and it’s entirely for this reason. I didn’t think it would be possible for anyone to actually do what needed to be done to get this technology across the line. That is to say, give it to customers before it is safe.
If Tesla is actually able to keep doing this without it getting killed politically, then I think it actually will happen, and that gives me some hope for humanity more generally.
> Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."
I think that what we've learned throughout human history is that no one should be trusted and given the power to make such decisions. It has invariably led to disasters. The very idea that someone can decide to put someone else's life in danger for potential future advances gives too many perverse incentives - chief of which, rationalization that "more people will be saved" even as the experiment crashes and burns.
When it's for the greater good, or by the winners, we conveniently omit it from history.
When it's for the greater evil, or by the losers, we publicize it as a cautionary tale of moral hubris.
Example 1: Vaccines
This is an interesting example because the winners and losers seem to be proportionately spread across almost all of society (except anti-Vaxxers who benefit from herd immunity but don't sacrifice with the minute risk of vaccines). See also: https://www.hrsa.gov/vaccine-compensation/index.html
Example 2: War on Crime / Safety
Another example would be the war on crime. Liberties and lives are sacrificed in the name of safety for others. Unfortunately, the liberty/lives sacrificed fall disproportionately into one group, and those protected fall disproportionately into a different group group. If you look at the 2nd order effects (impoverishment, single-parent households, effects on education and future) lots more lives are lost, again in a subset of society.
This is clearly possible in principle. After all, fully autonomous vehicles can have much more capable sensor systems than any human driver, can process the input from those sensors faster, and can then apply any necessary adjustments to the vehicle's behaviour almost instantaneously, all without ever getting tired or distracted. No human driver will ever match that potential.
The problem I have with this whole argument, and thus the reason I'm not sure the moral/ethical arguments stand up either, is that we don't know how to reliably write excellent software yet.
The entire argument for autonomous driving rests on the premise that we will at some point in the usefully near future reach a situation where the cars drive themselves much better than we humans collectively do. But there is, to the best of my knowledge, no evidence so far that we as a society can actually create the necessary software yet.
Until we can, any autonomous vehicle control system will be at risk of life-threatening bugs. In particular, it will be at risk of widespread failures of the same type that could cause much more harm than any single human driver's failure could, whether caused by defects in the software alone or by a malicious actor exploiting a security vulnerability. That is a huge risk that is getting far too little attention so far IMHO.
The first is whether we expect the autonomous vehicles to become safer than the population of human drivers they would be replacing under normal circumstances. This is a population measure, and it's about overall safety and averages, and given enough time and data it seems a realistic outcome.
The second is the risk of a single catastrophic failure, which is a risk that doesn't generally exist with today's human-controlled vehicles. It is no exaggeration to say that the kinds of autonomous vehicle technologies we are conceiving here have the potential, if they go wrong, to cause more harm than any WMD ever deployed in war. And not only the theoretical potential, but given the current state of our software development abilities, a non-trivial chance of it actually happening in practice.
Until we've dealt with the latter problem, which I don't believe we currently know how to solve, any arguments based on the long-term superiority of autonomous vehicles over the capabilities of any human driver is going to be fundamentally flawed.
You realize that those two statements are in conflict with each other right? If they're misleading and favoring Tesla then you need contrary stats to state that. However you state that they're not available, so your previous statement can only be a lie.
Any way you cut it, Autopilot and FSD are statistically worse than human drivers.
This is the only data Tesla has released comparing similar cohorts of vehicles (i.e., modern luxury cars). The only stats in which Tesla beats the average driver is where they include all cars ever put on the road, including semi trucks...but under those models every other modern car is also better than average.
[EDIT: And this is the Forbes article on Tesla's report. There's also a Cleantechnica post on the same report as well. https://www.forbes.com/sites/bradtempleton/2020/10/28/new-te...]
If Tesla FSD was 3x safer than human driving Musk would be bragging about it. The fact that he's not indicates that my interpretation is the correct one.
Waymo is at least as safe as the average human driver, while Tesla has unleashed literally worse-than-drunk drivers upon us all. Heaven help us.
They aren't just using their own customers, they are using all of us.
The cars won't be just crashing into inanimate objects, but school buses and ambulances. The QA is being done to all of us and our loved ones near their cars.
You are in between two walls and there are two straight lanes. Lane 1 has Tesla FSD coming in at full speed. Lane 2 has a human driver coming in at full speed. You can choose to be in one lane. The FSD and the human can both see you only 50 mts out and if they brake as soon as they see you, you will survive.
Which lane will you choose?
And FSD isn’t orders of magnitude safer. They are barely as good as human drivers and only in some conditions.
For the worst situations they aren’t comparable at all (since they won’t drive at all)
This must be repeated over and over for some reason: people will never accept self driving cars that are just as good or bad as human drivers. Nor should we. It’s a much too low bar.
Self driving cars will need to be orders of magnitude safer than human drivers to be even remotely acceptable.
I hate almost everything about Tesla and their business model but the idea of full automatic driving, once vetted to even a bare minimum level of approximate human parity, is enough to make me consider one anyway.
I'm having a hard time parsing your double-negative, but when shadow mode is running, it's choices are back-tested in real-time with the driver's choices, so I'm not sure how this fails to replicate what a real-driver would have done in such scenarios.
e.g. You see the driver swerve for no apparent reason, did the driver simply drive more recklessly than the car would, or did they avoid an obstacle the model didn’t catch?
People get easily distracted, get excited, get tired, all of which are propitious to bad decision-making.
Many human drivers are downright dangerous, but that's not a reason to add dangerous machine drivers on the road, either.
https://m.youtube.com/watch?v=xj73Q3lWvFM
It's been a problem for years now.
As a concrete example: if you assume that 99% of accidents that would be caused by the system will be prevented by driver interventions, and you think overall the system will reduce the likelihood of an accident by 20%, the question you have to ask yourself is: if 1 in 5 people harmed over the next 6 months we could have saved, should we avoid doing so and wait until only 1 in 200 will not be harmed, in part due to their own negligence? As you can see, the assumed probabilities matter a lot, so I'd be curious how one can come up with good projections. The only case where these probabilities don't matter is if you believe that engineers ought to never create harm where none would have existed otherwise, but when it comes to self driving cars, this is an impossibility since it assumes perfect autonomy.
In general, if you buy into the theory that "human + AI" is always going to be smarter than just AI, then arguably even in a world with great autonomous driving, the best system will be one that has both an AI and a human. And that is what we have here, so it's more a question of if AI quality is sufficient or if this is just a bad idea in general given the natural tendencies of humans to be poor co-pilots.
There's a term for it: "normalization of deviance"[1]. A highly risky thing is done and doesn't fail as early as thought. The people doing it assume that means the risk is lower than they thought, but it's not, and they just happened to get lucky.
I'm sure Elon knows this, so probably believes their tech can advance enough before their luck turns. I personally think he's deadly wrong on this, but it seems like a lot of Silicon Valley Tesla owners who sleep or read their phone while on Autopilot accept it at face value.
[1] - https://en.wikipedia.org/wiki/Normalization_of_deviance
I believe this is not entirely correct, as the parent talks about how a company can get away with something, while the term you use applies to the company itself.
The main question should be: why do we (the government) even allow companies to do something like this?
Stuff like that is only possible in the US. Would be unthinkable in the EU. Maybe in UJK but you know, they are no longer ...
1. https://europe.autonews.com/automakers/audi-bmw-others-frust...
This is analogous to banning airplanes until they can be proven safe to fly in. The countries that don’t ban them will lose more lives early on, but they’ll have a technological advantage. The place where this analogy breaks down is that hundreds of people are killed by human driven cars every day. Every year cars kill 120,000 people in the US & Europe. If we can develop and deploy this technology a year sooner, we can save tens of thousands of lives.
They already claim that drivers must pay attention to the road in order to prevent accidents and this has been the status-quo in the automotive world for a century or so. Selling products that can kill if misused is not remarkable.
This could turn ugly if it's shown that enabling self-driving results in much more accidents but there is no data to support this.
I don't understand this point of view. Cars WILL kill people with AI or humans driving. AI is not great right now, but it's already way better than humans in several situations.
Then people get spooked by bugs that the AI brings because they're not "normal" accidents, and think it's a bad idea to continue to train the AIs.
But why? Why is a "beta" AI killing people worse than the normal everyday drunk/distracted/bored/texting human driver?
Every stat that I've seen that comes out with the machines better than the people isn't a like-to-like comparison. Every like-to-like comparison I've seen has the machines still worse than people, so we don't even need to have that argument. Beware the propaganda.
If you take a shitty driver and compare it to an AI, I doubt the AI will perform worse when it comes to preventing accidents.
Sure, a skilled human driver will likely perform better than this early stage AI, but if even 1% of the bad drivers out there switch to an AI, that's a net positive in my view. Not to mention all the other risky situations involving alcohol, sleep deprivation, etc. where an automated response can save lives.
You didn't address my main point though. Why is a driverless accident worse than a drunk driver?
All car manufacturers sell products that can kill people.
Yes, that's exactly why most of them have much better QA than Tesla.
And where do the other people on the road factor into the decisions of you and your 5000+ lb vehicle?
Let's start "Falsehoods self-driving programmers believe about roads:"
All highway exits are to the right from the right-hand lane. If the left lane line curves away, the road is curving and you should follow it.
(1) None of the companies involved have ever had the motto "move fast & break things." Zuckerberg created that philosophy to Facebook. It's foolish to apply it an entire industry.
(2) They are no more "using its customers as QA" than pharmaceutical companies do when gradually ramping up trials that also kill people. There has been heavy testing before this point, and it seems entirely reasonable to scale up testing before a general release.
does it even become profitable before the 25 year mark in the current market?
In a way, it's good that the Tesla system sucks so badly. If they had a disconnect rate of one per month, drivers would trust the thing.
The other guys, disengagements per 1,000 self-driven miles in California:
Waymo, 0.076, or one per 13,000 miles.
Cruise, 0.082, or one per 12,000 miles.
Anyway, Tesla does not have "full self driving". They have slightly better Level 2. Not even Level 3. You'd think that by now they'd have hands-off freeway driving totally automated to a better than human level of safety. But no.
I have two driving modes, not autopilot, but not as aware as I could be, and then tracking everything I can because something odd is going on and I'm trying to avoid a bad scenario. Perhaps I'm a bad driver, but I suspect this is normal, kind of in a low-attention mode most of the time, but if the car a lane over starts to change lanes too close, you go into high-alert mode.
I wonder if there are studies on how often humans need to do this sort of "disengagement" for lack of a better term? Not sure how you'd measure it, perhaps with more eye movement than normal? Dunno.
Waymo just published an extremely detailed paper explaining exactly how many of their disengagements would have resulted in a crash, among other things. It's very interesting reading and shows how far Tesla has to go.
Summary: https://www.theverge.com/2020/10/30/21538999/waymo-self-driv...
The paper: https://storage.googleapis.com/sdc-prod/v1/safety-report/Way...
Tesla, 0.076 per 1,000 miles, or 1 per 13,219 miles
So you're comparing Apples to Giraffes right now, let alone oranges.
> In another video, Brandon's Tesla was making a left turn but wasn't turning sharply enough to avoid hitting a car parked on the opposite side of the cross street. "Oh Jeeeesus," Brandon said as he grabbed the steering wheel and jerked it to the left. "Oh my God," Brandon's passenger added.
Maybe it's just me, but this screams clickbait and low effort. The overall article is fine, but I nearly stopped reading after this. Is this kind of writing really necessary nowadays?
Not on HN usually.
> OOOOH MY GOOOOSH! - Data scientist REACTS to possible DATA BREACH by HAXORS
... I doubt it would help highlight the need for security. On a more general level, I personally don't think it's worth to trade reputation for attention via clickbait. But it might be the better option, I honestly don't know.
Imagine BuzzFeed would start to warn about a serious issue - sure, they get the attention, but with their reputation is going to be hard to get people to take their point seriously. And I hate to see ArsTechnica going in this direction.
The article is correct that expecting average drivers to do this without training is a high risk move. I've given flight instruction, taking over a landing 20ft off the ground is way harder than landing a plane yourself. And that's with a lot of training, not just an average driver being put in a place to supervise Tesla autopilot with no training at all.
Isn't that exactly the definition of cruise control ?
I'm sorry but I won't be driving a car without cruise control anymore: cruise control saves oil and is much more comfortable.
I imagined it was something you could enable and then disable by just breaking like cruise control or auto parking with toyotas.
But now I see how complex it becomes because of the wheel, even tesla owners in the comments of the article don't like to let their hands off the wheel.
Prime example would be the Uber test driver who felt she could trust the car enough to watch a TV show on her phone, when she was supposed to be supervising it.
Plus our attention naturally just starts to wander when the talk at hand is not that engaging.
Current Teslas allow Autopilot on the streets, it just tries to go forward on the intersections (with FSD it also tries to take turns). Stops on the lights and stop signs, btw. So the way you use it, you engage it on the long stretch, where it knows what to do, and monitor on the intersection. If it tries to take wrong lane on the intersection, you interfere. On simple intersections it goes through just fine. All this without FSD Beta.
Is it autonomous? No. Is it way less tiring to drive a car, when you don’t have to do lane keeping, speed control and traffic light stops yourself all the time? Absolutely.
For most people, lane keeping and speed control is ingrained enough they do it subconsciously.
That's not just theory, the article describes there have been 3 serious accidents already with "self driving" Teslas that crashed while being monitored by the driver.
Is that really obvious? It looks like the system is aware of much more than just lanes. A lane assist system doesn't stop for red lights or stop signs, or make left turns into a side road after waiting for oncoming traffic to clear.
Given that there’s regular stories of people playing phone games and crashing whilst using the current lane assist tech, are we safe?
‘Real-world testing was needed to uncover what would be a "long tail" of problems, he added.’
If it was actually just at their own risk, it would be substantially less bad. They opt in at their own risk, and at the risk of those around them.
While on the road I try to stay as far away from any tesla as I can.
Cynical view - it's being tested by twitter users who are on good terms with a company, with an explicit goal of generating more hype for Tesla.
1: https://storage.googleapis.com/sdc-prod/v1/safety-report/Way...
Disclaimer: I work for Google but am not involved with Waymo. Opinions my own.
Tesla does rely statistics on collisions while self-driving.
https://www.tesla.com/VehicleSafetyReport
> In the 3rd quarter [of 2020], we registered one accident for every 4.59 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot but with our active safety features, we registered one accident for every 2.42 million miles driven. For those driving without Autopilot and without our active safety features, we registered one accident for every 1.79 million miles driven.
I do think the counterfactual "how many times did the human driver taking over prevent a crash" data is really important to have. In the article, the scary near misses they talked about fell into that category rather than into the "actual airbag deployment" category.
It would be nice to have however I think it misrepresents. If the end result of having Autopilot on is that the driver, on average, crashes less then it's an improvement, even if the driver has to regularly intervene. What's important is the end effect, not how it happens.
> Tesla ... says it's not intended for fully autonomous operation. Drivers are expected to keep their eyes on the road and hands on the wheel at all times.
If tesla rolls out a product where you have to pay close attention and be ready to take over and calls it full self driving, that seems like a massive case of fraud. I'd be pissed if I'd bought the FSD package.
2. It's a beta.
3. People bought into it early when the price was lower. It's a subscription to the feature set as it develops, not a guarantee that it works perfectly the day you pay for it.
Not only because anecdotes carry more weight than statistics in the media—as this story illustrates—but because people are much less afraid of bad things happening, as long as they're in control when it happens.
Also, I suspect everything associated with fuel-burning cars is by now irrevocably linked with cultural ideals of individualism and freedom. Resistance to EVs and self-driving tech will continually materialize, as if out of thin air, regardless of how well documented the benefits are. It's going to be an uphill battle for the foreseeable future.
The reason I make this obvious statement is you are making it seem like the article is using anecdotes while there are statistics somewhere that show safety
Multiple drivers are reporting multiple incidents of varying severity in the short (week?) that the "full" FSD beta has been available. That's a huge concern, and it absolutely needs to be followed up on.
I don’t see a reason to believe that humans are so far superior at visual processing that no self-driving based on video could best them and many reasons to think that we’re inferior in certain aspects of the task.
Our road systems are not designed for the full dynamic range of the human eye, which is why we drive with headlights and we have street lights on highways rather than relying on night vision for all driving.
> The human visual system is backed by the equivalent of strong, general AI that intuits and understands the entire world around the vehicle and all objects interacting with it.
I think the Human general AI vastly overestimates it's ability to drive and respond to conditions that suddenly appear when driving.
> It is in theory possible for a video-only system to work, but ML is nowhere near the level of development needed for that to be a reality.
[citation needed]
1. https://www.engadget.com/waymo-indepth-details-selfdriving-a...
Taking a relatively simple to work around example, you need AI to distinguish lens flare from an object approaching/departing.
The "lane" in front of it is wildly moving around, so it's definitely misinterpreting the visual inputs.
In contrast, the animations of a car battery being charged don't claim to be realistic in any sense, and they deliberately attempt not to be realistic: oversized images of batteries or other standardized iconography, stylized arrows or other indicators of charging "flow", etc.
If you want to see longer FSD videos check out https://www.youtube.com/user/pilotjc78
I really don't understand how so many smart people are repeating this argument over and over. it's one the most flawed arguments that was used by Elon Musk to sell his lack of ability to retrofit lidars in every single car.
We also got a single brain, so why can a computer not do what we do?
I wouldn't like to be in a car that drives like either the Tesla or the Waymo cars though, the Tesla's are too aggresive, and the Waymo cars are not aggressive enough (which is generally good and safety aware, but would still be frustrating to ride in).
That's probably a reasonable response. These things have the nasty feature that they both require less input from the driver the more sophisticated they get, but also that they require more _attention_ from the driver the more sophisticated they get (a boring old cruise control system won't abruptly swerve into a concrete road divider, but a high Level 2 system will). And humans just don't work like this. "Don't do anything, but be ready to take control within a second or so when it does something insane" is not something we're good at.
Any level between that is dangerous, irresponsible, and should be banned from use on public roads.
Which leads to an interesting question or two: a) why isn’t it? b) what will happen to Tesla and Uber once it’s clear that “Unsupervised driving” will remain a decade away for at least another decade?
Reactionary regulation. Once a few high Level 2 driver assist systems drive into schoolbuses or something, expect to see that change.
https://amp.detroitnews.com/amp/26312107
My key takeaway is regulation of advanced technology lags waaaaaaay behind the disruption caused by the technology.
I thought regulation was something that you could define based on a sufficiently widespread practice.
The number of outright faulty actions I'm seeing on these videos is terrifying. Pedestrians and other cars on the road didn't ask to be part of this live fire experiment.
Yes. You can absolutely set safety metrics and boundaries. They might not be perfect, but as you get experience you refine those boundaries.
Just like settings alarms in web services -- you have a sense of roughly where the performance should be, and you set some alarms. You iterate on those alarms as you get data, but you still absolutely have a sense of what's roughly appropriate from the start.
Giving people the option to sacrifice their lives to contribute to solving this problem is an amazing thing when you think about how many million people the capability could save over the next century.
These are the risk taking explorers of the modern era.
Given that that's not how this is being marketed, I don't think the people "sacrificing their lives" would necessarily agree with that.
https://youtu.be/RN5Qoei7v1k?t=2218
It can't even figure out which traffic lights apply to it.
This system is really bad.
Look at how it responds to pedestrians here: https://youtu.be/RN5Qoei7v1k?t=587
And his quote: > Improvements across the board, turned into Target and took me to the front! Drove around East Sac, through Sacramento State, Midtown, and Downtown. This release definitely was far less interventions for me.
Also this guy in general gives a lot more vocal reaction than I would personally give in similar situation so it makes it look and feel worse than it actually is.
- How often do drivers intervene?
- How often do drivers fail to intervene and the system causes an accident?
- How often does the system take an action which has a high likelihood of having prevented an accident?
- How often does the system do so when it seems likely a human driver would have failed to do so?
- Integrated together, what's the expected dynamics of these probabilities and their net impact insofar as the system being releases more widely creates training data to help improve them more quickly over time?
It could very well turn out to be the case that this system is purely positive, strongly net positive, or neutral in harm reduction. The question then is, given that, what ethics should inform its release: is putting the stress on the driver sufficient if it say, saves 1000 lives in the next six months and will harm noone in exchange, other than some drivers having to intervene and have moments of stress? What if it will save 1000 and 10 people will be harmed by failing to intervene? What if it's an even swap, harming people who fail to intervene in exchange for saving people who would have inevitably been lost to accidents they couldn't have prevented?
Saying "this is wrong to do" based upon anecdotes is dumb analysis. Saying "this is wrong to do" based upon an absolutist form of ethics is fair, but it also means you reject that we should be trying to solve self driving. If you think it's wrong to do and are not doing either, you ought to articulate what scenario in terms of data would justify the action vs not, even if that data is unknown right now.
It could also turn out to be negative, right? Certainly the anecdotes shown here would point much more to that - for example, its inability to turn sharply enough is indicative of some deep flaws.
> If you think it's wrong to do and are not doing either, you ought to articulate what scenario in terms of data would justify the action vs not, even if that data is unknown right now.
It's wrong to do as long as you rely on CUSTOMERS to ensure safety. There is no further argument to be had: as long as the system is not PROVEN to be safer than a human driver even in the absence of human attention, it can't be put into the hands of customers.
That is not to say that they shouldn't be developing it, and paying and training people to operate it safely. That is what all other companies are in fact doing - taking a safe and responsible route, which is apparently too much to ask of the most highly valued car company in the world.
Saying "there is no further argument to be had" is just a way of shutting down debate. Why exactly is asking customers to opt-in to testing this somehow off the table? I'm not saying you're wrong, but you haven't given an argument and just are stating your belief as if it is some immutable fact.
Because they are likely to not just harm themselves, but also others. And also because they are not given access to enough information to be able to make an informed decision on whether the technology is safe to use, under what parameters.
Not to mention that the lack of accountability this creates for the company introduces perverse incentives - the company is getting free labour, and the more it hides about real stats, the more free labour it is likely to get.
This is all pretty well settled ethics - it is for the same reason that we don't allow companies to run beta tests for potentially life-saving drugs. This is long settled both legally and ethically, and the self-driving case is essentially identical, so why reopen that debate?
More importantly though, I believe that the statistical characteristics are more relevant here - I have no idea to how much risk I'm exposing myself by activating the Tesla "FSD", because (1) I don't know how much I can trust myself to switch from supervision to split-second control, and (2) I have no idea how badly this has been going for other drivers of the Tesla, or how likely the Tesla is to make really poor decisions (will it stop before the intersection, or plow directly into traffic? What percentage of the time will it do either?)
Extremely often, if the videos linked in the article are anything to go by.
Similarly, if you know interventions per mile, but don't know miles of challenging circumstances per mile, you can't compare two modes. Maybe people only turn self driving on in particular circumstances. Or maybe tesla opened the beta to people in easy locations. Apples to apples safety comparisons require a lot of detailed data when they aren't part of a randomized trial (which is still hard to do correctly).
Are we anywhere close yet? No.
Then why is Tesla calling it as Full Self Driving? Because it's marketing and more money can be made in updates.
Is it wrong to call something as Full self driving when it obviously is not full self driving? Well, duh!
But it is not a lofty ideal, it has been done before. And in the field of transportation. Trains used to crash all the time. Driverless trains don't. Why? Because fail safe systems have been added on railroads that nearly totally prevent this.
Don't take my word for it, the bar for rail safety is defined in well documented standards (namely, EN 50129, EN 50126 and EN 50128). It is 10-9 hourly probability of fatal failure. You can also look up "CBTC" on the Internet. This is the reference bar for driverless.
The real reason Tesla and friends don't want to go there is because fail-safe systems like a CBTC requires some form of collaboration with devices installed on the ground infrastructure, meaning Tesla should accept to become part of a system, and not "the" system. Obviously a paradigm change for the car industry.
How long will the public authority require to understand that they must step in to force a systemic collaboration and standards between infrastructure and cars to ensure driverless safety, I don't know, but I think it's time to call bullshit on the concept that you can have an acceptably safe driverless car system without it.
Absolute bullshit. 99% of drivers will frequently commit a series of small mistakes on an everyday <5km drive.
And he said "a series mistake", presumably meaning something resulting in a crash. The statistics back him, in that case (1 in 500k miles, someone else mentioned).
> Absolute bullshit.
It's not, though.
> 99% of drivers will frequently commit a series of small mistakes on an everyday <5km drive.
Perhaps, but the vast majority of such series of small mistakes don't add up to a serious mistake, so that's not relevant to the claim it was offered to rebut.
Kudos to the driver ++
I'm gonna be so fucking pissed if I die to Tesla's flawed risk/reward assessment, or worse yet, my kid. I bet Musk wont lose a moment's sleep over it though.
Since when did this ever stop companies when there are money to be made. Driving cars in general kills tens of thousands of people per year and so do many other activities. For governments it is just a question of how much money vs number of people killed.
From my point-of-view, the fact that TensorFlow completely scrapped its entire ≤v1.0 graph-based API for a PyTorch-style eager execution API for v2.0 onwards does not suggest that it was “designed from the ground up” in any way, shape, or form. We're talking a total rewrite of the entire frontend codebase going from v1.0 to v2.0, and a total abandonment of all v1.0 style code. TF 2.0 is basically a brand new language.
Under the hood it's all the same BLAS/LAPACK operations anyway. I've found TF superior to PyTorch for stats methods and probabilistic programming in general (far richer API), but for standard linear algebra the two are equivalent performancewise, with PyTorch having the superior API IMHO.
Even the fact of suicide by vehicle is complex: AVs can prevent some suicide by driver but possibly raise the risk of pedestrians committing suicide by vehicle. One reason suicide by vehicle happens is because it can conceal the intent to commit suicide. This is compounded by suicide being a taboo subject. This alone will complicate and possibly delay adoption of AVs.
Millions of innocent people are injured every year and about a million die every year in road accidents. Cutting that number significantly would have roughly the same impact on global well-being as curing a dread disease.
There are potentially hundreds of millions of new customers for vehicles as developing nations grow their middle class. If humans drive those vehicles, hundreds of thousands more will die every year.