Tesla FSD Beta Lunges Toward Bicyclist
twitter.com
twitter.com
His FSD and AutoPilot naming schemes, tunnel deathtrap, the FSD and AP actual driving errors, horrible Telsa customer service, anti-union and abusive workplaces, the whole pedo-sub-diver debacle, "Funding Secured" tweet, etc.
I did not agree to be in the beta experiment, and neither did that cyclist. This is playing with people's lives and someone is gonna get killed again, soon.
Will no one rid us of this turbulent priest?!
But still, even before this clip, I make sure to stay far away from Teslas if I see them out, they are just scary to me..
Also, a plea to other drivers… Please! Stay off the bumper of Teslas. The software is generally cautious and gets spooked. It rapidly decelerates a couple miles per hour in preparation for a full on evasion, which wouldn't be a problem for a driver with a sane following distance, but if you are close enough to angrily look down at my speed limit driving head through my glass roof things could get a little awkward.
It is not always safe to remove one hand from the handlebar to signal. In fact, in a lot of urban spaces, it's downright dangerous given the bad quality of the streets. Another reason why cyclists don't signal is when they notice a car too close or too aggressive. In this situation, it's best to have full control of the bike.
The cyclist is not a threat to your life. At most to the exterior of your car. You are a threat to their life. They don't want you to crash on them or the other way around. Give them space and it should be fine.
At least with the Tesla it's just going to kill you through its own stupidity...
But the only time that should happen is if there's an immediate danger that needs to be avoided. If we're going to have this awkward level 2 style self driving, it should at least be predictable, and preferably inform the driver what it wants to do well ahead of time (e.g. in this case, maybe it wanted to change lanes for an upcoming right turn..?)
I would definitely not risk using this thing in a city yet..
... or anywhere. Highway stakes are just as deadly, just less pedestrians and bicycles. Also, things can go wrong much faster there.
And yet Teslas plow into emergency vehicles stopped on the side of the road at a rate much higher than other cars.
It does not have to be this way, and these companies and their leadership deserve every bit of reputational damage they will get out of this.
In the end it's a regulation problem, not a Tesla problem (even though yes, the specific decision to exercise their options is theirs) - do you allow unlicensed drivers to perform this on-road testing of unfinshed vehicles or not?
It's not quite a deliberate information hiding campaign as with MCAS (as far as we know). But it is pre-production testing of experimental engineering on safety-critical equipment. Experiments not guaranteed to pan out, and engineering lacking a mature certification process.
"It might fire in random directions at arbitrary intervals 5% of the time, but that's just how innovation goes folks. You don't like that, do you hate progress?"
This is a thing. They give marketing samples to organizations they want to sell to on a "try 'em out and let us know if you want to buy more" basis.
Giving organizations stuff you haven't 100% nailed down yet expecting them to come back with stuff they want changed in order to buy is pretty par for the course in the firearms industry. That's as close to a public beta as you're gonna get.
It would be more akin to handing out prototype weapons to police departments, which doesn't happen specifically because they might fire themselves or blow up or fail in some annoying but not spectacular way like failing to feed new rounds.
At the end of the day the analogy isn't great and we're building a castle in a swamp here.
I assure you that 15 year old young adults drive much better than a Tesla "self driving" computer with millions of miles and millions of hours of training.
You’re a regular Yaroslav Kudrinsky, just with better luck. But I suppose one can go around swinging swords in a crowd, hit no one, and convince oneself of one’s good judgment.
The process of not wanting to impact something and attempting to avoid it can occupy so much attention that hitting that thing becomes inevitable. People tend to go where they are looking.
People are arguing, with a straight face, that we are ruining (aspects of) humanity by not letting Tesla be laisez-faire with such things (and lets be real, it's predominantly Tesla who is just in full Cowboy/YOLO mode... other players in this field certainly aren't perfect, but they are at least trying to be conservative and cautious).
As long as Tesla can demonstrate a similar or better rate of serious harm than human drivers, why should we stop them? It's up to government to do their job and outline exactly what price society should bear in the pursuit of this paradigm shift (because let's not kid ourselves, FSD is a complete shift in paradigm for society).
Because it can't, or won't. Instead they release statistics that _imply_ that it is better, that they have to know are absolutely misleading, because anyone who did high school statistics would be aware thereof.
Everything else, they refuse to release.
This is another major regulatory failure. Government should lay out the minimum data that the companies must publish, so we can get a better idea of their performance. (It's still a chicken and egg scenario though. You can't get that kind of data anywhere but on public roads in uncontrolled environments.)
I don’t think it’s a regulation problem as much as it is an acceptance by regulators that driving is inherently dangerous and autopilot isn’t dramatically worse. It’s not even obvious if on net more people would have died if Tesla had never released autopilot.
Did you delete your link to tesladeaths.com because you realized the fine print about sketchy autopilot data contradicted your point? Oops!
I am sure I have seen comment links without the http:// but might have just gotten so used to it than I stopped seeing it.
[0] https://jalopnik.com/feds-tesla-autosteer-safety-investigati...
I just don’t have data showing it’s terrible.
Further the only reasonable standard is roughly average human competency. As soon as a tired driver can turn on self driving and be safer on the road then banning that technology has a real cost.
The general public doesn’t use products like you might assume. For example toilet plungers have killed or seriously injured people and not because they where used as a weapon.
2. How many times has autopilot disengaged and would've caused injury without human intervention?
Note also that other drivers are learning behaviors such as keeping their distance from Tesla's, etc., making it harder to assess the system functionally.
Edit: Actually, there are docs for the scoring, sans changelog though: https://news.ycombinator.com/item?id=30267858
Are you joking? If you’re the one causing the problem, you’re the problem.
No regulation can prevent me from kicking someone in the shins. This isn’t a regulation problem, it’s a me problem.
The car problem presented here is quite a bit more shades of gray and we absolutely need to set acceptable parameters for the technology.
You don’t need a regulation telling you not to be a jerk — unless of course you are a jerk. And even then, you’d probably still be a jerk, regulation or not.
It's also very likely that this system can pass a standard driving test: AIs are really good at handling fixed test cases. So any regulation you do pass has a complicated assurance problem: how does it prove the system is safe?
Pretend Tesla just knocked "beta" off the name and said "this is a limited self driving system, Tesla exclusive" - then you wind up in the same bucket with the same "it's the drivers responsibility problem".
People see this footage and assume Tesla self drive does this all the time - but it wouldn't. It probably passes this scenario in testing 100% of the time: but the issue of course is the test is not quite whatever this circumstance is, and the system has a dramatic failure.
There is absolutely a regulation problem.
Can the software automatically make lane changes? Can it initiate turns? Can it decide to make non-emergency stops at lights and signs? Can it accelerate from a stop? What is the expected level of human intervention?
The question I'm posing is, how do you prove it works to an independent authority?
It would be extremely reasonable for us to regulate along these lines, but I'm also absolutely willing to bet that Tesla and every other system would get through an approval process just fine.
Which sort of loops around too a wider issue: if self driving is a priority, then it needs to be a government, collective priority for public safety and development.
I.e. independent reading institutes, a suite of diverse randomised tests and yes: because this is no longer "on the driver" for accidents then both the government and company will be taking responsibility for failures.
The issue is that the law of the land is: these systems aren't illegal, and the driver is responsible for how the car handles including any automation they engage.
To make any progress you have to break driver responsibility - or use that bright line and just outlaw the whole lot.
It would be Diesel Gate all over again. The testing would have to be documented somewhere and the car could just be trained for the test scenarios. Considering how often Tesla like to massage their numbers (nurburgring data, sales figures in basically every country, most recently Australia, the big asterisk next to their 1.9s 0-60 run, etc etc) I can't trust they wouldn't train directly for the test and neglect everything else just so they can have the best headlines once again.
In fact, there were licensure processes for this sort of thing already in place in California iirc, so this is less gray area, more "lets outsource testing to the end user".
Frankly, it's downright disturbing.
It's very apples to oranges here, but loosely relevant wrt/ information/documentation and in the cameras-vs-LIDAR/sensor fusion debate.
This makes it even harder to assess the safety of the system - other drivers are helping it along as well.
Then came Autopilot and later FSD, and Tesla threw the concept of safety was thrown out the window. An autonomous Tesla decapitated a man in 2016, and Tesla continued unabated! What actionable steps did Tesla take as a result of that incident to make sure it never happens again. Even today in 2022 you can find videos of Teslas on public roads just completely failing to detect and colliding with stationary objects. The culture at Tesla seems to not care this is happening, as it continues to happen.
It used to be that when your robot did a single thing wrong or caused the most minor of injuries, let alone it killed someone, then you engineered the hell out of it and implemented safety protocols to make sure that failure mode would never happen again. Not at Tesla, apparently.
I get the utilitarian argument that maybe self driving cars could reduce traffic accidents in the future by being better than humans on average. But safety doesn’t happen unless you really work at it, with intention. Using your customers to beta test your 2 ton robots on public roads is not a safety-focused mindset. Tesla is not a company that values safety first when it comes to their autonomous car project.
- SCADA always had hot, hardware backups and failure detection. To the extent that I thought this was dogma.
- Physical automation (like car construction equipment and robotics, pick and place machines, etc) have safety zones that humans aren't allowed in or near. Same thing for Amazon warehouse robotics.
I'm not sure how we leapt to, "beta test cars on the street" from there.
Everyone who has ever written code for a robot knows you should treat every pinch point as a finger blender, but the cars look nothing like them.
Tesla's branding may make that excusable for the layman, but for the employees at Tesla, that branding is incredibly irresponsible
This keeps being touted (by Elon and others), but is a long way away: https://twitter.com/Tweetermeyer/status/1488673180403191808
Tesla uses misleading stats (like not excluding highway miles, which are inherently safer) to try and make it seem like autopilot is any good.
I keep harping on it but the decapitation incident in 2016 is just so illustrative of what kind of culture Tesla is fostering. Here's a situation where reckless engineering choices by Tesla literally caused a customer to lose his head. The solution which would fix this problem would be to add more orthogonal sensor modalities, so that any blind spot in one sensor can be compensated for by another (e.g. LIDAR will not see a glass wall, but SONOR will).
But what did Tesla do? They actually doubled down! They removed existing orthogonal sensors and they made everything worse! Now basically the entire thing is driven by vision, and it's still predictably exhibiting the same failure mode of running into broad and stationary objects as it did 5 years ago (which by the way was approximately the amount of time Elon Musk predicted it would take for them to get to level 5 autonomous driving. Instead they've bumbled around for half a decade and failed to make any significant progress).
Kind of almost 1 meter, ( 3 feet) wide.
So it thinks the tiny patch between the bollard, and the right most white line, is the right driving side of the road.
Many of “us” are already dying, and it saves lives in the future.
/s
- Beta tests critical software in the public which has lead to actual customer deaths;
- Refuses to follow industry best practices when it comes to safety sensors;
- Refuses to seriously investigate safety incidents and implement technological and cultural fixes to prevent them from happening in the future;
...will lead us into a utopia of automotive safety. At best what they're liable to do is create a climate where mega corporations feel free to use public spaces as their own personal laboratories, because the ends justify the means after all (don't pay attention to the decapitations, we're doing it for the safety of us all). We will all pay the price for that.
- willfully misrepresents data to imply a better safety record than they actually do have
https://twitter.com/Tweetermeyer/status/1488673180403191808 (I've posted this elsewhere on this thread too)
Tesla's self-driving beta is unethical because most of the participants in this "experiment" did not opt in. Most of the people killed by Tesla won't be their customers. It will be the cyclist or the pedestrian that gets killed because the driver thought "full self driving" meant the car was fully driving itself. They didn't even know they were part of an experiment that day, yet they were the sacrifice that Tesla was willing to make.
Just so we're talking about the same term:
> The belief that all individuals, or nearly all individuals, can succeed on their own and that government help for people should be minimal.
https://www.dictionary.com/browse/rugged-individualism
Typically, the rugged individualists are the ones who say that the government should stay out of all this and people can take care of themselves. Basically the opposite of what I'm saying.
Neither ethic would approve of systematically killing people for the greater good, but they're two very different ethics in most other respects. For example, I could arrive at "vaccination should be mandated" from a sanctity-of-life perspective, but I never could from a rugged individualism perspective.
For Tesla, though, FSD is still such a front and center thing that I'm having trouble seeing how it ever plans on saving face as it gets more and more bad press from incidents like this.
Remember that guy in southern California who got arrested for sleeping in the back of his Tesla while it drove him home, and he said he did it many times and would do it again? The system works well enough that he did it many times before he got caught.
That guy is an idiot, but think about that story from another angle: You can fall asleep while driving any car. The difference is, if that car is a Tesla, you might not die.
As a driver aid for non-crazy people FSD is a good system and Tesla owners like it. It's far from being vaporware.
Based on the videos I’m seeing of FSD in action, I have my doubts. Wouldn’t be surprised if this was a publicity stunt. Elon Musk has a very loose relationship with ethics and the truth. They’re just obstacles for him.
Right. Similarly, the fast Prime delivery guarantees from Amazon are backed by warehouse workers working their asses off, and I'm sure we've all heard about the bad rep Amazon gets for worker treatment and the subsequent attrition challenges they've been facing.
It feels like Tesla is setting itself up to be stuck between a rock and a hard place, being reliant on their FSD marketing to make money but simultaneously tainting their brand beyond salvation by this very same reliance.
Amazon doesn't seem to be hurting either, despite the bad press they get about their warehouses.
People are some of the worst drivers around, they cause all kinds of problems with cluelessness and inattention and they crash and die and kill and injure in all sorts of stupid ways. They get drunk, high, check their phone, fall asleep, speed, show off
I think it's great that there are still companies out there throwing research into moonshot types of research projects like this. Automobile deaths are perhaps the biggest and most persistent pandemic we've faced, we've been facing for a hundred years, and it kills and maims young and old alike. Millions of people dead over the years.
Many were willing to relax some of the protocols around vaccine testing to bring the covid vaccines in to use earlier, put mandates and coercion around them, and yes some suffered adverse reactions and even died from them. Still, it was thought to be a reasonable compromise by many.
I hope we don't go the other way on other technology that can advance society and potentially save lives just because not everything goes perfectly from the start, or we get stuck in the pharmaceutical corporations = bad mindset.
Not to say there perhaps shouldn't be more regulation around it, but a serious and careful study of risks and benefits I think is appropriate rather than handwringing and outrage.
We don't need new technology, we need new leaders.
You're not the first one to have thought it's a good idea or that we should try to do something about it. It's not like it's easy and people have just decided not to.
Compared to which other species?
1 fatality per 100 million miles driven is relatively good, to be honest.
This seems to me like a good argument for the public funding of electric rail lines instead of self-driving cars that perpetuate all of the problems personal vehicles are responsible for.
Handwringing about someone carefully testing a car and not hitting a cyclist in the process is really pretty rich. I'll happily take actual data and statistics but for every one of these I can also post a video of a driver doing something stupid that a computer would not.
A 1 out of 100 to 1 out of 1,000 fatality rate for COVID depending on demographics is quite literally several orders of magnitude higher than a 1 out of 100,000,000 fatality rate for driving. You're comparing apples to atoms.
So no, it's more like apples to watermelons. The watermelons being bigger, the ones which have killed far more people and will continue to kill far more people.
So even COVID-19 wasn’t enough to “move fast and break things” (though I remember hearing many such calls to “move fast and break things” from the same contrarian-sphere community that is now Very Concerned about vaccine injuries). If COVID-19 wasn’t bad enough to completely remove our safety guardrails, certainly the never ending promise of FSD (watch any Musk video compilation where he promises FSD is just around the corner, going on five years now) isn’t enough.
Edit: I just looked up the numbers and it’s 102 deaths/day average in the USA. I think it’s a horrific problem and support many initiatives that will take cars off the road and calm traffic.
102 deaths every day in the USA alone is staggering, and many children and young people! Are you trying to it's not? How many people has self driving cars and their research and development killed? Vanishingly few by comparison.
The situation is very analogous to covid and vaccines.
The guardrails that the typical trial process has, see e.g.,:
https://coronavirus.jhu.edu/vaccines/timeline
https://www.ncirs.org.au/phases-clinical-trials
> AFAIK none were removed, it's just steps were done in parallel at great $expense.
No, the far greater expense is the typical 5-10 years that clinical trials require.
> It appears you are trying to imply that it's fine if Tesla skips safety testing and pops a few skulls like watermelons in pursuit of $FSD. I'm not saying perfection is possible, but certainly Tesla can do better than it's doing. If anything, Tesla threatens to set everything back with its sloppiness. One bad accident could create enough bad PR to set self-driving cars back for years.
It doesn't appear that I'm trying to imply that at all, it appears you're just making things up.
Again, you’re claiming vaccine guardrails were removed but posting no evidence. Do they do 5-10 year trials on the annual flu vaccine? Please show me an authoritative medical source concerned about major shortcuts taken bringing the mRNA vaccine to market. Should be a piece of cake.
I don't know what you mean by guardrail either. Vaccine trials were significantly accelerated from typical vaccines, long term data was foregone and emergency use authorizations were used to deploy them. I don't know what you're trying to argue or whether you're actually trying to deny that reality.
From other communities I could buy the "fear of the unknown" bit. But here, most of us are involved in the production of software in some form. It's not the unknown that we fear, it's the known. We fully understand what modern software is actually capable of, and we know that it's not ready for prime time in these kinds of environments.
Further, many of us have been in a chaotic public beta before, and seeing those patterns play out during a beta test of lethal machinery is really scary. Rapid chaotic iteration is fine when it's a simple SaaS product. Not so much with Tesla.
Furthermore, no self-driving system will be perfect even in a mature stage of development. People will still die, and they may die because of mistakes a human driver would not have made.
If society isn't willing to accept that sacrifice, then let's just ban self-driving cars.
But let's not try to have it both ways -- let's not allow self-driving while also holding the view that risk of death is unacceptable.
That's equivalent to a ban.
>If entities want to innovate, let them innovate within safety parameters.
Sure, but the safety parameters have already been established[0] at 40,000 deaths a year, or less, for transportation equivalently convenient and functional as compared to a manually-driven car.
[0]By the crowd-sourced wisdom of democracy.
Are you saying that, despite transit alternatives that do not lead to excess deaths, that deaths due to self-driving cars is worth the cost? We have existing solutions that don't result in these excess deaths, but because of some reason, we don't want those solutions _and_ we want to kill more people just to birth this technology? _Why_?
> [0]By the crowd-sourced wisdom of democracy.
... Which bill are you talking about? I don't remember my Rep or Senator voting on allowing me to be a casualty of the self-driving industry. The fact that automobiles in the US result in 40k deaths is itself a travesty. Even Canada is doing better than the US here. Pointing to the worst driving safety record in the G8 and one of the worst in the G20 isn't exactly proof positive that the American automobile system is safe, it's just proof that we have grandfathered an extremely unsafe system here.
That's just not realistic, not even in Europe.
Why wouldn't it be realistic?
If a state government or the US Federal government tried to ban cars, they would be out of power very quickly, even in the most liberal parts of the country.
The existing solutions <As chosen by the collective will of the American people> cause 40,000 dead Americans every year. Other 'Solutions' are not. Don't ask me why they are not, ask your neighbor, and your congressman -- but they are not. Also ask yourself how many collective minutes spent waiting for a bus is a traffic fatality worth? This isn't rhetorical, the answer is a number.
>Which bill are you talking about?
Lets start with the United States Department of Transportation, followed shortly thereafter by your state's department of transportation.
>it's just proof that we have grandfathered an extremely unsafe system here.
All the reason why a potentially new system should have to meet the safety standard of 'less unsafe' and not 'perfect'. Literally every day the adoption of self driving is delayed kills 109.6 people[0] because that's one more day of the status quo. To save lives we should be damn near throwing people in front of Teslas to help them train instead of wringing our hands over the matter.
[0]As much as, divided by the relative eventually safety of a fully developed self driving. As Americans, more globally of course.
Americans choose automobiles because of zoning laws mandating massive setbacks and minimum parking requirements. There's nothing in the DoT or my state's DoT that mandates use of a car. It's all due to American city planning, which is driven by auto friendly city planning policy. So again, why is transit out of the question here?
Self-driving cars aren't an incremental improvement away. We have no idea how many more deaths are needed to mature the technology. So why are we bending over backwards for it when there are proven solutions to move people without casualty? If self driving can innovate without casualty, then be my guest.
You would kill tens of thousands of people a year with that policy decision.
The unstated assumption in this is that self-driving is feasible with current technology and without enough training will be better than human drivers.
All of the companies that are testing self-driving cars have had accidents with other cars and pedestrians. Some of them resulted in death. Others did not, but easily could have, and eventually their luck will run out and they will also cause deaths.
Who would invest millions of dollars into developing a technology when a single mistake could flush all that money down the drain?
Sensible legislation is needed to balance innovation and safety for self driving cars on public roads, but not zero. We don’t even accept zero pedestrian hit and run deaths yet…
California, Texas, and Michigan legislators have already bought in to the benefits that these companies will bring.
It's not like Tesla has to release new features every year, they just feel they need to to continue to pump up and justify their stock price.
EDIT:
I misread the parent comment, I’m not opposed to just banning self driving cars as self driving cars likely won't make everyday folks' life better.
Just because this instance was regarding a bicyclist doesn't mean it's only a danger to bicyclists, there are still ways for people to die if the car were only interacting with other Teslas.
I'm more of the mind that cars should become the second-class citizens they ought to be in cities and that I should be able to bike around the city without constant threat of death, that dream remains far off in american cities.
Why is that fire codes and aviation regulations are written in blood, but traffic violence is treated as a normal, necessary cost of modern life? When a biker is killed by a driver who wasn't signaling and cut them off on a right turn (as happens to me on a daily basis in SF), that should trigger immediate redesigns of intersections and streets across the city, for that was an entirely preventable death.
But nope. Go ahead and test your fundamentally flawed, unproven death machine on public roads.
I gotta say, I am really not that keen on being killed by a white tesla.
What rules did Tesla break?
Also, remember that time Waymo killed a pedestrian?
waymo never killed a pedestrian
Yeah, it was Uber, my mistake.
But that's an example another self-driving company that killed people. It could have easily been any of them -- they have all had accidents that could have resulted in death, and just got lucky. Tesla is not special here.
Previously: https://news.ycombinator.com/item?id=30182141
Generally speaking, the average human driver causes a “ding” every 100k miles, a insurance worthy event every 250k miles, a police worthy event every 500k miles, and a fatality every 60M miles. Assuming an average driving speed of 60 mph, which is obviously an overestimation, those are 1.6k, 4k, 8k, and 1M hours respectively.
Evaluating a L2 autonomy system is somewhat challenging since in truth you are evaluating a human+autonomous system, so you should expect the overall safety to increase unless the autonomy system is actively making things more dangerous. However, for a L4/L5 autonomy system, which is Tesla’s goal, we can reasonably approximate the quality by miles per intervention as an intervention is a failure of the autonomy.
As a reasonable baseline based on video evidence of FSD, we can assume that each intervention prevents at least a “ding”. Therefore, we can evaluate the quality of FSD relative to a minimal L4 system as the ratio of time per disengagement versus 1.6k hours. Using those same videos as evidence, we see the FSD system requiring intervention every few minutes, i.e. <10 minutes, on average. That is 1/10,000 of the minimal quality expected from a fully functional L4 system.
So, for Tesla to create a minimal acceptable L4 autonomy solution that would actually usher in a age of safe self driving systems, they need to improve the core functionality of their beta product by 10,000x. It is that gap between what they are delivering and what they still need to achieve that is outrageous.
In the mean time, we could start to get a better handle on the safety profile of these systems by better classifying the interventions.
Something like these:
* Intervention due to unreadable conditions (e.g. unpainted lines, a new area with no training data, failure of LIDAR / camera, w/e)
* Human intervention due to not feeling safe
* Intervention due to unsafe condition (e.g. imminent collision, or just hit something)
* Human intervention for other reason (e.g. human is bored, or human changing destination, or human wanting to take another route)
---
#1 could certainly lead to #2 and #3, but is more of an indicator that the car is in novel conditions not trained for, or has hardware failure. #1 is the low hanging fruit (relatively speaking), and #2 is where the majority of the work lies.
#3 is where the system has actually failed.
And #4 need to just be discarded.
> we can assume that each intervention prevents at least a “ding”.
I don't think we can be sure that the vehicle wouldn't have corrected itself in time, nor how many would have been much worse than a "ding". This is where some closed course testing with simulated pedestrians and bicycles etc may give more evidence. In evaluating self driving software, knowing what the consequences of a failure are is important. As you point out, safety drivers prevent this assessment.
> based on video evidence of FSD
I think that there is selection bias in the readily available footage. Firstly, the short clips are likely to be where there is an incident. The longer ones are usually beta testers deliberately choosing tricky routes to test the FSD. I don't see anyone posting videos of a 3 hour freeway drive where nothing happens.
But my point is just that this is complex, and we should be much more careful than we are being.
However, self driving is definitely guaranteed to fail if it can't be tested on public roads. And that's really the challenge here -- it can only succeed if society is willing to take the risk.
It's to be expected that people will occasionally die in car accidents. Some things are hard to avoid. If someone on a bicycle comes shooting out into a busy street when they don't have the right away and no one could see them coming, they're probably going to get run over. My general expectation for self-driving vehicles is: don't create unsafe situations that weren't there before, don't behave in unpredictable ways, and if you can't avoid running someone over at least do it in a way that the consequences fall on the person doing risky things. In the above example, swerving to miss the biker is fine if it can be done safely, but swerving to miss the biker and instead running over someone on the sidewalk instead is not. (Or, in other words: if you can't avoid an accident, at least be predictable and try to obey the general principles of right-of-way.)
Swerving at a bike for no reason as in the video above should result in suspension of self-driving features (assuming the video is real and wasn't staged in some way).
I can see this being a reason to ban a system from operating at SAE level 3 or above, but isn't the entire difference between level 2 and level 3 that things like that are expected to happen at level 2 and below, hence the reason for drivers having to always be ready to immediately intervene at those levels?
Honest question: If beta testing speeds up the transition to FSD by x months/years and causes y deaths during the intervening period but FSD ultimately reduces the number of road deaths by z per day. For what values of x, y and z is it an acceptable trade off?
Zeroism is often overly simplistic. y being non-zero might be hugely net-positive in the long run.
Even if FSD is mature and 1000x safer than human driving then there will still be deaths. Are you suggesting that if FSD is 1000x safer then it still shouldn't be allowed on the road because there is the possibility of deaths?
I'm not claiming that FSD should or shouldn't be allowed on the road right now. My point is that we shouldn't wait for the chances of death to be zero.
This is a solved problem with experimental design. Experiments are designed and reviewed by ethics boards, and participants are gathered after getting their informed consent and giving them just compensation for the risks involved.
"Testing of FSD solutions in the real world" can follow the same path in their experimental designs, but they don't, because executives and shareholders don't want to pay for expensive ethical experimentation. They'd rather move fast and break bodies.
Most modern life saving devices and treatments have gone through exactly this type of ethical experimentation. There's no reason Tesla's products couldn't, as well.
“Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.“
everyday driving comes down to that same acceptance of risk
Let's not be perfect be the enemy of the good: a self-driving highway system can probably be achieved with current technology that beats human drivers for long distances. I wish that was the priority rather than the robotaxi obsession.
Self-driving highway systems would produce immediate profit and economic savings that could fund the drive-grandma-to-Taco-Bell systems, as well as the convergent infrastructure, common signaling systems, embedded sensors, better signage and standards, improved and predictable construction signals and routing, etc.
Following the usual hype cycle, it will be highway logistics that smooths over the "despair" portion and does the practical incremental improvement.
Of course, many people will interpret FSD as a convenience feature (as they do for “autopilot”) and will neglect to properly monitor it.
I've heard it described as being like a driving instructor, watching a very inexperienced student who mostly drives fine, but is liable to suddenly make extremely poor decisions.
What I'm seeing here is more like teaching a chimpanzee how to drive. It's irrational, has no clear intent, and does not ever announce what it's trying to do.
I've never been in a FSD-beta car, but I would take a novice adult driver over that every day of the week.
Part of the difficulty of trying to have reasonable discussions about FSD is that so much of it is black-boxy nobody really knows what's going on. I don't care if Tesla wants to keep its ML models proprietary, but I feel like we deserve to know how it uses those models to make driving decisions on public roads. And not just Tesla either; every self driving company should be required to open source their "dumb" code so that we understand them better. The competitive advantage comes from the ML models anyways, so it wouldn't be anti competitive to regulate that disclosure.
"data to understand fault and liability following a collision must be accessible;
[There are] sanctions for carmakers who fail to reveal how their systems work"
The proposals are very detailed but do seem to be addressing a number of areas around how autonomy is marketed, and how to ensure vehicles are safe and that vehicle makers are making available material information on the inner workings of their software.
It’s like GPT-3. In 9 cases out of ten, it responds in a way that lets you think the model has “grokked” facts of the world, and a human like sense of logic. In truth, it hasn’t. It’s just pretending. It’s taking the easy path by parroting back what it has seen before.
They say that ML requires a re-thinking on the part of the developer. Whereas with traditional programming it would be his job to “structure the problem and the path to its solution”, in ML, he should hold back his human notions and let the network discover its own structure.
To use an analogy: Whereas we teach our children concepts step by step - you and me, identifying objects, gasping objects, referring to objects, etc., all the way up in complexity to writing thoughtful comments on HN - the neural network is to be bombarded by unfiltered real world data. And the hope is that, if you just do this long enough, the AI will come up with the important underlying concepts by itself. That is to say, for GPT-3, the idea that words are really describing a real world out there, with objects that have certain properties, and that claims may be true or false. Or, for Tesla, that the car is a physical object in space, and that it can collide with other objects.
This is very unintuitive. It might be the right approach. But if I was building a self driving car, I would want to build that system block by block. Learn the basics. And only proceed to the next level of complexity once I have verified that the previous step has been properly learned.
1. Let the car learn to predict it’s own acceleration, steering and braking. 2. Repeat on different road surfaces. 3. Add other objects to avoid. 4. Train ability to tell objects from non-obstacles (fog, raindrop on camera) 5. Train lane markings, traffic lights on empty roads. ... X. Throw real world data at it. Now you can hope (and validate) that the model will have the foundation to make sense of it. Like a kid after Highschool. It will still make dumb mistakes. But at least you can talk about them and understand why they happened and take corrective action. Because, while you do not know the other’s mind, you do speak the same language.
With GPT-3 and with FSD, we just have no idea.
So if an incident like this happens and we don't hear anything from Tesla it tells me there's a far, far bigger problem here with their failure to collect diagnostics, or failure to analyze these problems. Ideally we should see a hyper detailed root cause analysis even going into the vision model inputs in this exact case and exactly how they failed to classify and detect this cyclist, and most importantly how the beta software is being improved to fix issues like this. Silence is deafening here.
Its sensitivity was probably lowered because it would brake "too often" for a comfortable consumer product if it was sufficiently cautious enough to avoid hitting pedestrians.
This is what happened with the Uber self-driving vehicle that struck and killed a pedestrian[1]. The system detected the pedestrian as an obstruction, but Uber had programmed the system to ignore it because of "phantom braking". Instead of braking, it proceeded to strike and kill the pedestrian.
It’ll be interesting to see if they ever add LIDAR/re-add radar and admits their mistake.
I'm extremely skeptical they're going to fair much better on scale up: its notable that the Uber crash had LIDAR and still hit a similar object (a metal mesh with lots of gaps and a person).
after all, who would want to bike somewhere with no protected bike infrastructure that allows shit like this to happen on public streets?
Sure, some environments like extreme weather are not being entered by most self-driving companies. However, their detection stacks in general work fine in the vast majority of places, it’s their decision making that needs work.
As others pointed out, Uber failed not because of a lack of detection of the person. Moreover, the cases where Tesla publicly fails aren’t particularly challenging for detection (daylight, good weather).
But we all (should) know that's not how machine object detection systems work: they would reliably pass this sort of scenario all the time, and then some sort of adverse (input data wise) input situation from something humans don't even notice would trigger a complete failure.
Which is to say it's not intuitively obvious that other systems won't have these sorts of failure scenarios - it certainly won't be proven till they have sufficiently widespread deployment to eek them out.
It seems like these issues could be easily avoided by putting LIDAR back in. I'm afraid that they're hesitant to do so because they'd have to admit that removing it was a mistake for all of the cars on the road without LIDAR.
*: HD maps are mainly for complicated interchanges, visualizations, and aid in not curbing the rims of the car; a few early FSD videos show what it does when it really has no map data, and it's less confident overall in turns as it tries to make out the edges of a road.
It has to be watched really closely and each build seems to replace old problems with new problems.
I love the idea of fully autonomous driving where I can sit back and read a book or watch a movie, but I cannot imagine what it would be like driving a car where you are not actually driving, but have to be constantly on your guard in case it decides to do something unexpected.
Driving with constant attention is fairly easy, I would imagine not driving, but having to continually monitor ready to make split second would be exhausting.
but seriously, that was a serious WTF video. The comedic timing and the horror of what that car just tried to do.
It is worth noting that humans make stupid mistakes like this too, but I don't think this excuses the automation's mistake. Cars don't abuse substances (okay, most of the time), so we should be throwing out like 75% of all accidents right there. Do better, Tesla. Beta testing like this is unacceptable when it is fucking with cyclist's safety.
On the technical side of things, I can see how a slower-but-still-moving object next to the car might screw with path-making. But this isn't how the bugs should be hammered out. I say we outlaw all OTA updates for vehicles. If it didn't ship with the software it's running, it's not street-legal. (And as for vulnerability fixes, Tesla's smart. Let them figure it out.)Humans generally don't. Only humans that are impaired (drunk/lack of sleep) do this more than "autopilot" in its current state: https://twitter.com/Tweetermeyer/status/1488673180403191808
How large is the biggest fake city used for soak in testing by one of these autonomous car companies?
How much did it cost?
These Tesla threads are always tense but there’s a lot of important discussions to be had about Tesla and FSD wrt technology and society. That said user: omedyentral seems to be farming anti tesla rage both here on HN and on their Twitter page. I’m not sure that’s in the spirit of what HN is trying to be.
I feel this. I rarely if ever use autopilot because I find the cognitive load and stress of monitoring it to be higher than just driving the car myself. The exception to this is when the road is absolutely empty.
There are anti-{FB, Google, Big Tech} posts on HN all the time. I’ve never seen anyone say that’s against the spirit of HN. What makes Tesla special?
It's actually your post that is against HN guidelines[1]:
> Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.
I don't know what the FSD license says, but I'd look into Lemon Laws in your state. If any other part of your car made you have legitimate concerns about your safety, you'd take it back; same thing should apply here.
Tesla had a good run with this contraption, but it is time to scrutinize this fraudulent system that does not work as advertised and malfunctions on the public roads to run over or crash into cars and cyclists.
This is not even at night that FSD has malfunctioned, let alone night vision support. The car doesn’t even monitor the drivers attention properly (It still uses the wheel). How can FSD be possibly safe to use when not only the system has failed in daylight but it also cannot see at night?
Perhaps it is better to not pay and sign up as a crash test dummy with safety-critical software that doesn’t even work.
No thanks and absolutely no deal.
The Northeast has high density, old infrastructure (narrow bridges, tunnels and highways, lack of medians and breakdown lanes, etc which are out of spec to modern standards), fewer sunshine hours, cold weather (and the resulting pot hole & lane line degradation of the streets)..
On the other hand, I keep seeing a lot of crazy FDS videos from the Bay Area so clearly they don't even have their home turf covered.
I still don't see how local road FSD is going to happen on current hardware - camera resolution, placement, and CPU inability to handle high frame rates.. We are perpetually 5 years away.
Seems like Google, Cruise and co are doing well enough here to make it look like a tesla problem as opposed to a self driving car problem.
The responsibility is still on the driver even with driver assist / full self driving technology turned on. All this talk of regulating the software when we just saw a worst case scenario resolve with a favorable outcome.
"But what if the driver wasn't paying attention?" Then the driver will face the consequences as if he had been driving the car himself, because it's his car and it's his responsibility.
In Australia until you were off your probationary license (which was a minimum of three years after your learners permit, which was either 1 or 2 years, it's been a while), there were power-to-weight restrictions on the vehicles you were allowed to drive.
Here? Saw a YouTube video of a seventeen year old doing his driver's license test in a 1500hp Bugatti. That'll end well, no doubt.
I liken Autopilot / FSD more to a horse at its current state. You keep a firm but gentle hand on the rein and let it find its way. Sometime you have to check it.
If you get complacent (which the screen explicitly warns you against), yeah, things could go wrong. That's also 100% true when driving a normal car.
Edit to add: I think a lot of criticism focuses on the term "Full Self-Driving" which implies "Full Self-Driving". That's fair, but I think it misses an important point. ("Autopilot" made a lot more sense to me: real autopilots can't do everything either. They can't even taxi.)
When you're actually IN THE CAR, enabling this functionality for the first time, you have to enable three separate toggle switches. Each one pops up a progressively longer, more dire WARNING dialog, emphasizing the software's primitive state, the importance of paying attention and keeping your hand on the wheel. It also demands your constant attention while engaged.
The thing that most of these companies are failing to grasp is that the whole endeavor is like 99% corner cases. It's corner cases all the way down. The hope was that there was this big chunk of the problem space that could be handled in a tractable way, and that would hopefully account for ~80% of driving conditions. Then corner cases could be handled on an as-needed basis to fill in the gaps, and slowly you could handle more and more scenarios.
The problem is that the reason for so many corner cases in this domain is that driving is a social activity. The rules of the road actually belie how much communication between humans goes on when they are driving. Driving has its own norms, its own language, culture, emergent behaviors, and traffic exists as a crucial force that must be balanced in a city ecosystem. But autonomous cars aim to solve the driving problem by ignoring the social, and reducing it to an optimization problem. Autonomous cars are blind to the larger world around them except to recognize colors and the nearness of physical bodies.
They miss social cues, they lack higher order reasoning about traffic patterns, they can't negotiate with pedestrians or other drivers, they are unable to weigh the ethical considerations between when following a law is actually worse that breaking it.
As of now I consider them menaces on the road, and that pains me as someone who got to play with some of the first driverless car technology in the lab. I had such high hopes for it, and what I hoped was not... this.
Did you miss the part where the autopilot steered straight toward a clearly visible cyclist?
If the autopilot is so unreliable that it requires lightning-quick reflexes to override, it's going to lead to a lot of accidents from people who don't happen to be paying perfect attention at the time. I know the whole story is that drivers are supposed to be 100% engaged, but realistically that's just not going to happen when you literally market something as "full self-driving"
The airlines that got fucked by Boeing's MCAS still had to send payouts to families who lost loved ones, because it was ultimately their decision to buy and operate a 787 MAX. They can (and certainly should) sue Boeing for the losses, but it doesn't change the fact that the crashes are theirs to own. You don't get a get out of jail free card for being gullible.
That's the best possible outcome, no?
This time...
> Presumably this incident is an important bug that will now be fixed.
What gives you this confidence? Tesla started decapitating people in 2016, and 5 years later it's still lunging toward cyclists, running over pedestrians, running into parked cars, running into barriers, running stop signs...
Tesla has a fundamental perception problem. Teslas fundamentally have blind spots due to their perception stack. The way this gets better is that Tesla:
- admits its perception system is a bust, and it goes for a more robust system which takes into account over a decade of industry wisdom as to how to properly engineer a potentially life-threatening system with safety-first engineering practices
- starts emphasizing a safety-focused company culture that doesn't use the general public as a beta testing playground.
I predict neither of those things will happen.
Unfinished vehicles on the road are nothing new; all companies developing cars move pre-prod hardware across public roads. However it usually requires several levels of additional licenses and special safety driving training and instruction before you're allowed to operate a car that may surprise you by behaving very different from a certified production road car. And regular renewal thereafter.
I think what's unprecedented here is the scale of the pre-prod testing, the type of experimentation being done and how the drivers are selected. And also the degree to which it's designed into the approach of the experimental engineering being done that it must fail to improve (by widening the training set). There's enough new here it should be discussed imho.
Side note: German driver's license training now includes some material on assistance systems and how they may behave unreliably and how to prepare for their failure modes ... but of course this doesn't retroactively expose existing license holders. The rate of change in driver-car interaction being so high is also a novum.
https://www.tesla.com/support/safety-score
1. what it uses
2. how it's calculated
So far they've let everyone in with a 98 or above, but they come in waves so getting 98 doesn't necessarily mean you'll soon have access.
This should be highlighted. I work at another car company, and we're not even allowed to drive the very lightly modified test cars without a special license, and those are essentially production vehicles just with extra debug logging. That extra license requires being CPR certified as well as two separate driving course certifications. The idea that Tesla can push this out to basically anyone who wants it is absolutely absurd.
No, what we should expect from a beta is for it to not be deployed at scale, to non-professional drivers, in a live environment. It's clearly still not ready to go live, much less under the name "full self driving"
Can we be sure that the average driver who has received no special training in the matter will be able to take over from an autopilot in a split second when driving at high speeds? Heck people can't pay attention and keep their eyes on the road even when they are the ones in control.
> The responsibility is still on the driver even with driver assist / full self driving technology turned on.
If/when the system is rolled out at scale will drivers actually comprehend that?
> Then the driver will face the consequences as if he had been driving the car himself
Assigning blame after the fact is fine, but no one can bring a dead pedestrian back to life.
The beta system controls for this by only allowing in people who score 98 or above in Tesla's safety surveillance program. The restrictions are tight enough that you have to actively try to be a safe driver to get anywhere close to the cutoff (some people even got pissed off because their score went down because someone brake checked them and they had to decelerate too quickly). Not everyone who buys FSD gets into the beta.
> If/when the system is rolled out at scale will drivers actually comprehend that?
The only ethical way to roll this out at a wider scale than the scale allowed by the safety surveillance program is if it's good enough to avoid incidents like the one depicted here. I think we can all conclude that Tesla deserves the fat lawsuits if they were to green light FSD in the state it is in today.
> but no one can bring a dead pedestrian back to life.
An unfortunate fact of life, hopefully made irrelevant by the comeuppance of autonomous transportation technology (including but not limited to automobiles) that makes motor vehicle accidents a thing of the past. Until we get there, though, someone has to keep trying.
Autonomous transportation already exists, it already works, and it's _already safe_! It just isn't deployed for private automobiles, it's deployed on rail-based transit systems. The technology is widely deployed across Europe and Asia. How many more people need to die to let Americans have a more comfortable ride in their private vehicles?
That's the entire point of America. We have so much space, so we make every effort to use it, regardless of the consequences.
Holy smokes. Did we watch the same video?
> Is this not what we should expect from a beta?
Is it OK for a beta software product to nearly kill another person that has no agreement with the company putting the product on the road?
People have the right to do whatever they want with their own stuff, fine. But putting others at risk comes with huge responsibility, and must be weighed against the desire to do what you want. In this case the balance clearly falls IMO to Tesla being selfish and negligent by employing their drivers to be the safety backstop in a clearly buggy system.
What consequences? Drivers almost never face any consequences whatsoever for manslaughter or murder with a car.
Even if they did face "consequences", they would not be anywhere near the consequences a completely random, innocent, and highly vulnerable road user will.
You must be unfamiliar with the usual consequences of a driver killing cyclist(s)
But cyclists rarely receive the protection of laws that supposedly exist for that purpose.
The issue here is that the person on the street does not know they are the ones likely to get impacted by a bug/failure of a software in beta. Beta products are usually not released to the public, or are opt-in.
The general public accepts the fact humans may cause fatal mistakes while driving, because responsibility is expected and ultimately there’s someone to punish. But automation making fatal mistakes, even at a lower rate than humans, isn’t accepted the same way, in part because companies are not individuals – Elon Musk isn’t personally responsible for any accidents – and in another part because this space is currently unregulated – unlike in aviation where some body has to vet a manufacturer’s plane project before it can be mass produced.
So, Tesla wants to iterate a product with significant risks on the wild while unregulated, at the same time enjoying a lower level of accountability an individual holds while driving – they just put on the contract that “self-driving” feature has to be monitored all times and if it fails the blame is on the driver.
This is not a surprise considering Musk apparently extremely libertarian views, but is obviously BS as it’s a way to dodge risk while capturing all the upside. It’s a perfect rational thing to do in his opinion (we’re taking risks to innovate, etc), but it’s ultimately unethical.
Maybe self-driving could be more reliably achievable with the help of local government adapting roads so cars don’t have to rely only on their sensors, or the whole industry could agree on a standard so cars can coordinate – but that would mean regulation, and Musk doesn’t believe in that. He believes Tesla can hack their way out of the problem only with technology in computer vision. I don’t think it will succeed, because if you look at aviation, another industry that enjoys a high level of automation and safety, nothing would be possible by initiative of a single plane manufacturer – the automation relies on equipment installed on planes, airports, etc. all standardized, as well as training and processes followed by the entire industry.
Time will tell who’s right, but my bet is a reliable self-driving car has better chance of coming out of a place in Europe, Japan, Korea or China, where manufacturers and governments have a better chance of collaborating to achieve a solution that isn’t stuck into the “computer vision” local maxima.
It’s like how there are thousands of gasoline fires a year but stop the presses if an EV goes up.
It's because EVs are new, and people want to know the risks. Granted, it would be nice for reporting to also mention that gasoline fires are as or more common, and or state the distinctions between fires from a gas vehicle v/s an EV.
Because Cruise isn't stupid or reckless enough to (a) have open betas and (b) deliver FSD at all without some form of human backup.
No offence to Tesla owners but too many of them don't take the limitations of FSD seriously enough.
Just two passengers in the back seat.
With vision, you're basically having a machine learning model guess where the object is, its size, etc. So there's a chance of failing the basic task of knowing that there's something possible to hit.
In other words, no, we don't ignore MCAS causing the plane to go into a stall under some isolated circumstance just because humans also put planes into stall.
Please translate.
Treat the Tesla FSD as a human driver and it would've had its license revoked many years ago. It's only through close supervision by humans that accidents have been as infrequent as they have been.
The clip in question, if a human hadn't been paying attention, FSD would've knocked a cyclist over - at best. Do that enough times as a human - not many - and you'll be disqualified from driving pretty promptly in most civilised countries.
No single human driver could do this amount of driving, so comparing all FSD mistakes to a single human is an error.
A lot of people simply don't realize that human drivers kill each other by tens of thousands per annum in the US alone, so there's a lot of room for self-driving technology to do a lot of good.
A lot of people are unhappy at being used as unwitting beta testers simply by virtue of having to share the road with beta FSD systems. They have a point, but see the previous one.
Finally, a lot of people question some of the engineering practices at Tesla, such as their refusal to incorporate multiple sensor inputs as failsafes. I'd fall into that category, personally.
There's a lot of room for self-driving technology that actually works to do a lot of good. Tesla's FSD is hot garbage.
Cue up those who think that it's fundamentally impossible to see a downside to anything Tesla does without having a vested financial stake in their failure.
That's categorically false.
Or are you going to claim that many of the people here in this thread, for but one example, are "Short TSLA"?
I'm not. I hold no financial stake in TSLA.
> A lot of people simply don't realize that human drivers kill each other by tens of thousands per annum in the US alone
According to the EMR system at my job, I've been an EMS responder on 378 fatality MVAs, and several thousand more non-fatality. And I still hold my position that Tesla is reckless. Throwing fixes over the fence for safety issues (phantom braking for one recent example) with 48-72 hours since the last update proves beyond any doubt that there's no ethos of thorough testing.
While you're right in the general case, the presence of "$TSLAQ" in the original tweet means that the author of it almost certainly does have a vested financial stake in their failure.
Is that really a good-faith interpretation of my post? Please consider reading it again, the whole thing this time.
In your answer, yes, you do address those things, but (with apologies if that's sincerely not your intent) those are lightly brushed off with a "but people kill in cars, too".
What is not brushed off, and is the first response you have, is entirely about people who have a vested financial interest in the failure of FSD, Tesla, Musk, or some combination of the above. And that's the only response you don't brush off with a "well, but". For but one example, in our marketplace, for better or worse, short selling is an entirely valid tool (though I'm not particularly a fan, and don't even start me on naked short selling).
So given that it's the first response you have, and it's a common refrain, "oh, people just want to see it fail", then it's not necessarily unexpected that I might express frustration at the discussion not paying much mind to the technology itself.
But as said, if that's not really your intent, my sincere apologies.
Huh? Yes, we do?
Of course, I think the solution to this is improving public transport and more aggressively taking away people’s driving licenses, rather than giving autopilot a pass.
My understanding is if you accumulate a certain number of points, it gets taken away. The reason DUI etc. receives such harsh penalties is because the number of accidents DUI has caused makes it obvious that drinking is a factor. It should be the same with "autopilot".
If you link a twitter video from a driver swerving erratically at cyclists, there will be no shortage of a mob demanding the driver lose his license and be put behind bars.
In this case the driver is an AI, so you respond by banishing all the cars this AI controls, goodbye Tesla FSD.