Scenarios in which Tesla FSD Beta 9.0 fails
twitter.com
twitter.com
If I drove like Tesla's FSD seems to based on the videos I've seen, I'd be pulled out of my car and arrested on (well founded) suspicions of "driving while hammered."
After a decade of work, it's not capable of doing much beyond "blundering around a city mostly without hitting stuff, but pay attention, because it'll try to hit the most nonsensical thing it doesn't understand around the next corner." It drives in a weirdly non-human way - I've seen videos of it failing to navigate things I'm sure I could get my 6 year old to drive through competently. Except, I don't actually let her drive a car on the road.
I'm out in a rural area, and while "staying in the lane" is perfectly functional (if there are lanes, which isn't at all the case everywhere on public roads), there's a lot of stuff I do on a regular basis that I've not seen any evidence of. If there's a cyclist on a two lane road, I typically get over well over the center line if there's no oncoming traffic to make room. If there is oncoming traffic, typically one person will slow down to allow the other car to get over, or the lanes just collectively "shift over" - the oncoming traffic edges the side of the road so I can get myself on or slightly over the centerline to leave room for the bike. And that's without considering things like trailers that are a foot or two into the road, passing a tractor with a sprayer (they don't have turn signals, so be careful where you try to pass), etc.
If they've got any of this functionality, I'd love to see it, because I've not seen anything that shows it off.
At this point, I think we can reasonably say that it's easier to land people on the moon than teach a car to drive.
Also out in a rural area. Running out to pickup lunch a few minutes ago, a young man flipped their old pickup truck on its side in an intersection, having hit the median for some reason. I too don't understand how humans are allowed to operate on public roads. Most of them are terrible at it. About 35k people a year die in motor vehicle incidents [1], and millions more are injured [2]. Total deaths while Tesla Autopilot was active is 7 [3].
I believe the argument is the software will improve to eventually be as good or better than humans, and I have a hard time not believing that, not because the software is good but because we are very bad in aggregate.
[1] https://www.iihs.org/topics/fatality-statistics/detail/state...
> I believe the argument is the software will improve to eventually be as good or better than humans, and I have a hard time not believing that.
I find it easy to believe that software won't manage to deal well with all the weird things that reality can throw at a car, because we suck at software as humans in the general case. It's just that in most environments, those failures aren't a big deal, just retry the API call.
Humans can write very good software. The Space Shuttle engineering group wrote some damned fine software. They look literally nothing like the #YOLO coding that makes up most of Silicon Valley, and deal with a far, far more constrained environment as well than a typical public road.
Self driving cars are simply the most visible display of the standard SV arrogance - that humans are nothing but a couple crappy cameras and some neural network mush, and, besides, we know code - how hard can it be? That approach to solving reality fails quite regularly.
Silicon Valley isn't going to suddenly become less arrogant, but that doesn't mean the problems they attempt to solve don't need solving. Incentives are important to coax innovation in a way that balances life safety with progress, and I concede the incentives in place likely need improvement.
We've had good automation in aviation for decades. It doesn't handle anything else in the way very well at all, and while there's some traffic conflict avoidance stuff, it's a violent "Oh crap!" style response, nothing you actually want to get anywhere close to triggering. Automated approaches down to minimums require good equipment at the airport as well as on the airframe, and if there's a truck on the runway, well. Shouldn't have been there.
Same thing for landing boosters. There's nothing it really has to look at and understand that it can't easily get from some basic sensor data - speed, attitude, location, etc. It's an interesting challenge, certainly, but it's of a very different form from understanding all the weird stuff that happens in a complex, messy, uncontrolled 3D ground environment.
Self driving on a closed track is a perfectly well solved problem. Self driving in an open world is clearly very far from solved.
The company website mentions the system was able to negotiate the challenges of the driving conditions, looks like an euphemism...
Very few details, no scientific publications I could find on their Asimov system on a quick search, not sure if this is a technological breakthrough or a fine tuning of existing processes and methods.
Because if its a fine tuning of current algorithms not sure how long they can push this. They are relying on Lidar (and other sensors) but even as of last year, it seems most teams already realized Lidar would not be the solution for Snow, and were now pushing Ground Penetrating Radar ( sounds expensive...)
"Autonomous Cars Struggle in Snow, but MIT Has a Solution for That" https://www.caranddriver.com/news/a31098296/autonomous-cars-...
Even Tesla, already realized its not just a sensor problem, solving self-driving needs a higher level algo than can put the different sensors, in context of situational awareness. Note that in no any other way I would consider Tesla an example to follow ;-) And do not think they are any closer to getting a working system. Some of the statements show at least a second level understanding of what is required. Sensors are a means to it. Its about situational awareness but also inference.
"LIDAR is a fool’s errand… and anyone relying on LIDAR is doomed. — Elon Musk"
I’m actually shocked it’s on the website today, first frame says the driver is only there for legal purposes.
Self landing rockets are simple, it's not rocket science, duh! Everyone and my grandma has one.
"Where am I, and what's between me and my destination?" isn't the hard part, as it is with surface driving.
The question has never been about the feasibility of landing the booster stages - what has been questioned is whether it's worth doing. The fuel used up during landing is fuel that cannot propel the payload. The landing might fail. The effects of thermal and material fatigue are not well understood. The transportation, refurbishing and QA are unlikely to be cheap anyway.
No, but I am not so sure it's anything more than a stunt with a high chance of failure.
>Launching reusable space launch vehicles that boost back to the landing site arrogance?
Maybe, but it's mostly a PR stunt from my point of view.
Just my 2 cents.
Safety-critical software like that used in the space shuttle is incredible expensive for the level of complexity involved (which is not very high, compared to other projects). A self-driving car is probably one of the most complex software projects ever attempted. If you were to apply the same techniques as the shuttle to achieve self-driving you would literally never finish (not even the tech giants have enough money to do this). So to achieve this you not only need to solve the very difficult initial problems you also need to come up with a way of getting extreme reliability in a much more efficient way than anyone has achieved before.
Self Driving cars (Tesla; who is faaaar from it, among others) will kill people. But people are shitty drivers on their own; has to start somewhere and Tesla is the first to get anything close to this level in the hands of the general population (kind of, beta program is still limited in release))
But logically this doesn't really follow, does it? That because humans are not capable of doing something without errors a machine is necessarily capable of doing it better? Your argument would be more compelling if Tesla Autopilot logged anything like the number of miles in the variety of conditions that human drivers do. Since it doesn't, it seems like saying that the climate of Antarctica is more hospitable than that of British Colombia, because fewer people have died this year of weather-related causes in the former than the latter.
It could be applied here to test if we are getting closer or further from what humans can do.
Some examples:
Smoke Test 1:
Driving with Snow
Smoke Test 2:
Driving with Rain
Smoke Test 3:
You are driving to a Red sign and you notice 50 meters ahead a pedestrian has its headphones on. The pedestrian is distracted and looking at traffic coming from the other way. You notice from the pedestrian gait and demeanor its probably going to cross anyway and its not noticing you. So you instinctively slowly reduce your speed and keep a sharp eye on its next action.
Smoke Test 4:
Keep eye contact with a Dutch cyclist as they look at you across a roundabout. You realize they will cross in front of your car, so you already inferred their intentions. Today the cyclist bad humor will not make them raise their hand or make any other signs to you other than an angry face. You however already know they will push forward...
Smoke Test 5:
A little soccer ball just run across your vision field. You hit the breaks hard, as you instinctively think a child might show up any second running after it.
Failing any of these scenarios would make you fail the driving license exam so I guess its the minimum we should aim for. Call me back when any AI is able to even start tackling ANY of these, much less All of these scenarios. :-)
slecht humeur? That's "bad mood".
See second definition: https://www.merriam-webster.com/dictionary/humor
The problem with Tesla's approach has always been that Elon wanted to sell it before he actually knew anything about how to solve it. It's lead to series of compounding errors in Tesla's approach to AVs.
The vehicle's sensor suite is woefully inadequate and its compute (yes, even with the fancy Tesla chip) is woefully underpowered... all because Elon never really pushed to figure out where the boundary of the problem was. They started with lane-keeping on highways where everything is easy and pre-sold a "full self-driving" software update with absolutely no roadmap to get there.
To overcome the poor sensor suite and anemic compute, they've boxed themselves into having to invent AGI to solve the problem. Everyone else in the space has happily chosen to spend more on sensors and compute to make the problem computationally tractable, because those things will easily get cheaper over time.
I'm fairly convinced that if Tesla wants to ship true FSD within the next decade, the necessary sensor and compute retrofit would nearly bankrupt the company. The only way out is likely to slowly move the goal posts within the legalese to make FSD just a fancy Level 2 driver assist and claim that's always what it was meant to be.
I mean they trained pigeon's to guide bombs in WW2.
Even wanting to do things vision only is ok, but the cars simply don't have enough cameras for redundancy. You are a little rain or mud away from having a system that no longer operates.
At this point it's pretty obvious none of the original "FSD ready" cars Tesla shipped will be able to handle L4/L5 autonomy hardware-wise. Arguably none of the cars currently rolling off the line will be either - others have made the point in the thread: while nobody has really cracked autonomy fully, other players (see: Waymo, Cruise) are clearly much further along with a very different sensor and compute suite than anything Tesla has shipped or currently ships.
L4 autonomy is where the vehicle is fail-safe. It cannot handle all conditions, but is able to recognize conditions it cannot handle and hand control back to the driver. Importantly if the driver is not responsive to takeover the vehicle can put itself into a safe state (mostly just means pulling over in a safe place). L4 is the minimum at which you can be fully distracted from driving.
I don't think Tesla has a viable path to L4 using its current hardware...
Wasn’t there recently a medical testing company that completely blew up, with associated felony charges, for attempting this sort of strategy?
Once we take proper inventory of where we are at, we may find that we are still so far off the mark that we would be inclined to throw away all of our current designs and start over again from (new) first principles.
The notion that you can iteratively solve the full-self-driving problem on the current generation of cars is potentially one of the bigger scams in tech today (at least as marketed). I think a lot of people are deluding themselves about the nature of this local minima. It is going to cost us a lot of human capital over the long haul.
Being wrong and having to start over certainly sucks really badly, but it is still better than the direction we are currently headed in.
As a model Y owner, I'm quite skeptical about this generation of vehicles being able to truly hit the mark on FSD.
"Just because we can put a man on the Moon does not mean we can put a man on the Sun" :-)
Try calling out a random diver a 'pedo', or perform the most cynical kind of market manipulation (then laughing to the SEC at their face) and your outcome will be very different.
It's Animal Farm all over.
Elon doesn't get any special treatment. You can do all these things as well if you're willing to expend resources when faced with repercussions. My suspicion though is society will give you more slack if you dramatically increase access to space for your nation state or make credible progress against a global ecological problem humanity is facing.
> It's Animal Farm all over
Don't get the Animal Farm reference as we're not talking about some sort of proto-communist utopia. Everyone is playing in the same sandbox.
Step 2: Publicly lie about some purported acquisition with the purpose of manipulating the stock of said company.
We can talk after you've done that, from jail probably.
You're neither an insider of said companies, nor are you a public figure with enough influence to actually manipulate the market.
A random dude made a post on reddit about how he planned to invest all of his meager (in comparison) savings into GameStop stock. The post caught on, we all know what happened, and he ended up being called by the SEC, accused of market manipulation, among other things.
https://freespeechproject.georgetown.edu/tracker-entries/sec...
The SEC is giving Musk free reign because they know that he can't leave the company because the two entities are so intertwined. Musk wealth is effectively just paper wealth.
The SEC doesn't care that investment bankers consider it real enough to give Musk loans for those pledged shares
They also don't care that the cult of personality he managed to create enable him to constantly not produce results and investors still give him money
In the end the SEC cares about real cash leaving the company coffins and into the owners pockets , not paper wealth swelling.
It's really a M.A.D. game between Musk and the SEC at this point...the SEC is willing to bet that it's not remotely possible that Musk lied through his teeth this whole time just to get to #1 in the paper wealth Forbes list only to implode while at the top.
Musk on the other hand doesn't strike me as a cold calculator such as Gates or Brin or Zuck, he is much more impulsive and there are non-zero chances that he did just that.
You can find a nameless CEO guilty. No one cares. No popular Twitter account or quirky pop culture aspects! But Musk knows his place he has created in our culture has made him untouchable.
Not just any random diver, a diver who had just helped rescue the lives of several children and adults from what was an internationally known emergency.
If any of us had said what Musk said about the diver, we'd be rightfully dragged through the mud.
It's important to point out that this software is currently only offered as a private beta to deliberately selected testers. Now, maybe they shouldn't be using it on public roads either, but at least it's not available to the general public.
Nine versions in, I would expect ongoing challenges with things like you mention. But continued failure to even see large, flat obstacles is no longer something that needs to be fixed – that it has persisted this long (even after killing someone as in the case of T-boning a semi trailer at highway speeds) is an indictment of the entire approach Tesla has been taking to FSD.
I used to think FSD was just a matter of getting enough training for the models, but this has changed my mind. When you still require a disengagement not to negotiate some kind of nuanced edge case, but to avoid driving straight into a concrete pylon, it's time to throw it all out and start over.
Completely mental that they are allowed to run this on public roadways.
Remember, with radar, Teslas will happily autopilot themselves into a semi truck across the road, stopped fire trucks, concrete road barriers, etc. I see no reason to believe that a stationary concrete pillar in the middle of the road would be avoided any better with radar than without.
https://news.ycombinator.com/item?id=16722461
Radar does detect stationary objects, come on.
Tesla's implementation of their radar processing drops all returns from anything stationary, given the forward speed of the car. In the context of a Tesla thread, I didn't think that needed to be specifically stated, but will do so in the future.
Emergency breaking on a highway when going under an overpass is also a huge problem, so radar may not be the answer.
Elon himself?
Autopilot v9 itself does not use radar data at all[1], additionally cars manufactured from earlier this year on do not even have the radar modules[2].
Note, that I didn't say "threw out the data" in a silo, I said "without having a comparable reliability in place with the vision only."
[1]: https://twitter.com/elonmusk/status/1397347380874203136
[2]: https://www.theverge.com/2021/5/25/22453518/tesla-vision-rad...
Who needs radar right? A few cameras are enough to discern gray concrete structures in the night, oh wait ...
> in the night
Purpose-built cameras can see better in the dark than humans can.
And yet...
Define "see better".
Human eyes have way more dynamic range (30 f stops) than even the best cinema camera sensors (14 f stops), and I highly doubt the sensors Tesla use are anywhere close to the gigantic expensive sensors used in cinema cameras and are most likely closer to the sensors in your mobile phone (11 f stops).
So no, no camera sensor can currently match the human eye, let alone the image processing and cognition our brains do on top.
Second, diverting the conversation from low light performance to dynamic range is an unimportant tangent. Deciding that one sensor needs to cover the entire illumination range rather than multiple sensors each simultaneously capturing farther in each direction than we're capable of is an arbitrary unimaginative choice.
Boeing 737 Max had only one AOA sensor [1], and that wasn't a great idea.
https://www.cnn.com/2019/04/30/politics/boeing-sensor-737-ma...
You want the best sensor type over a high-fidelity sensor and a lower fidelity sensor. The Tesla system has 8 cameras (3 forward), so they def have overlap between what they are considering better cameras.
Time will prove which approach wins.
> camera sees air, radar sees wall, slam on brakes
Seeing bridges as walls is not a fundamental property of radar. That's an implementation problem. If cars are doing radar poorly, maybe the fix is to start doing it less poorly instead of throwing it away entirely.
So back to LIDAR.. same difference. Camera and LIDAR have different profiles. I think it's fine to use either, but I think trying to blend the two is a sub-optimal solution to the problem.
Again, this is my guess from what I know. I could be wrong, and the winning technology could use 12 different sensors (vision + radar + lidar + smell + microphones), and blend them all to drive. Cool if someone pulls it off! But if I had to do it myself or place a bet, I would put it on a single sensor type.
Here's why:
> If your eyes tells you the floor is safe, and your nose smells something bad, do you stop?
Absolutely, yes, if the bad smell smells like dog shit or vomit or something else that I definitely don't want to step in. If I'm walking, I'm very unlikely to be looking directly at my feet and much more likely to be looking ahead to do predictive path planning. I definitely do stop at least transiently in your scenario and then apply extra visual focus to the ground right in front of my feet so that I don't step on dog shit. The center of my vision is great at discerning details, but peripheral vision is terrible for that.
Anyway, the obvious answer to your inquiry based on my explanation here is to use confidence weighting and adaptive focus. If I think something might be happening somewhere, I focus my available resources directly at the problem.
well, if I'm smelling gas, I know the situation isn't safe… (and thus my nose is giving me an information my eyes might not have detected)
Research over the last decade has shown that LiDAR/Vision fusion outperforms Vision Only.
Can you explain the science behind your position ?
If my eyes tell me the floor is safe, but my ears hear it creaking and my feet feel it sagging I'm going to stop in my tracks/back up. If Tesla can't handle correlating multiple different types of sensors they aren't ever replacing a human driver.
and yes, your percept is constructed from all functioning senses. what you look at and where your attention lands is often directed by your auditory system which can locate things in 3d space.
That's simplifying the situation a bit too much. The camera can give more results than air/not-air. Specifically in this case it could detect a bridge.
Same applies to the radar really - you'll get measurements from multiple heights which would tell you that it may be an inclined street, not a wall.
To be truly redundant you have to use different technologies, such as camera and lidar.
Right, which means it's not a solution for L4/L5 autonomy, only for L2. Tesla is trying to reach L4/L5, so just alerting the pilot is not satisfying the design goal.
> To be truly redundant you have to use different technologies, such as camera and lidar.
I think that is an opinion and not a fact. Watch a video such as
https://www.youtube.com/watch?v=eOL_rCK59ZI&t=28286s
from someone working on this problem
Neither is ignoring it. If a product can't meet the design goals under certain circumstances should it ignore it, not even look for it, or alert the user that there is a catastrophic failure?
> I think that is an opinion and not a fact.
I think it is more common sense than anything else.
This isn't necessarily true. From a reliability engineering perspective it depends on the modes of failure and the probability of each mode. If the probability of an AOA failure is low enough, you can reach your designed risk level by having two identical and redundant components. It all comes down the level of acceptable risk.
Uber seemed to have programmed an artificial delay when the system got confused. There's a good breakdown of the timeline showing how the system kept misclassifying the bicyclist who was killed, but I couldn't immediately find it. That breakdown shows their strategy at implementing delays in the decision process. According to the NTSB report[1]:
>"According to Uber, emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior"
When I read that in the context of the programmed delays it seems to indicate "we wanted to avoid nuisance braking so we put in a delay when the system was confused." As someone who used to work in safety-critical software, it blows my mind that you would deliberately hobble one of your main risk mitigations because your system gets confused. While TSLA may be focusing on a platform that gets better data with better sensors, they still need to translate it to better decisions.
[1] https://www.ntsb.gov/investigations/AccidentReports/Reports/...
Maybe it was put in place to avoid erratic braking in absence of obstacle, in order to avoid getting hit in the rear by other vehicles (whose driver wouldn't see any obstacle and be unprepared for the Tesla car braking).
The way this analysis is done is that a hazard gets assigned a risk score based severity x probability. I’m saying the severity of hitting a pedestrian is higher. Put another way, if you had to choose between being struck as a pedestrian or rear ended as a driver, which would you choose?
Just a small nit-pick but it makes the case against Boeing worse. The airframe had multiple AOA sensors but the base software only used one sensor reading. Note the image in [1] shows readings from both a "left" and "right" AOA. From your link:
>software design for relying on data from a single AOA sensor
Boeing sold a software upgrade to read both AOA devices. (This still leaves the problem that if the two AOAs disagree there might be cases where you don't know which is bad). The fact that they listed MCAS as 'hazardous' rather than 'catastrophic' means it was allowed to have a single point of failure. It also means they may not have fully understood their own design.[1]
[1] https://www.seattletimes.com/business/boeing-aerospace/black...
This should never have been greenlighted, approved, manufactured, designed. This went against a culture of safety (least surprise, etc) at every level.
I think "lying" may be too strong of a word, but they were certainly incentivized to believe it was just a modification of an existing airframe. If I remember correctly, Boeing was essentially told by an airline that if they don't come up with a design quickly the airline would instead move their business to airbus. Certifying a modification to an existing design is much faster than certifying a brand new design, so every management decision was made with that lens. Add to it the political side where the competitor is a non-US company and it gets more complicated. Add to that the FAA is likely too understaffed to provide adequate oversight so they instead delegate it to the manufacturer.
This will be another case study in misaligned incentives and the tradeoffs of engineering decisions under uncertainty of both business and design risk.
It goes eyes-shut-closed full-throttle balls-to-the-walls against safety. You don't rush nuclear power plants, why are we doing it with planes? (And yes we should use reuse already approved, known, standardized - modular, components. But they should be indistinguishably the same, not "we certified it the same by introducing so much complexity, so many new failure modes, that we have to hide those".)
And if safety costs a lot, then we can work on improving our processes (design, safety evaluation), but the process itself should lead to better stuff, not probably-not-that-worse but marginally cheaper so we can keep our semi-vendor-locked-in clients.
I'm aware of the Boeing-Airbus US-EU cockfight, and that as usual a coordination problem is at the root of this. But the way forward should be a healthier market, more competition, not a monoculture. (Or duo.) Or, if costs force us into such a pathological state of the airplane market, then we should treat it as such. (It should be viewed as one big problem. The total lack of healthy competition must be factored into the approval process. And a high-level Market Authority should step in to prevent regulatory capture and the usual - almost guaranteed to happen - shenanigans.)
My experience in aerospace is that the industry swings back and forth on this like a pendulum. A bad event happens and then there’s a focus on safety improvements. Then years or decades go by and no bad event happens. Rather than attribute this increased safety to the process changes, those very same changes are looked at as a burden? Why are we spending all this money and time on quality and safety checks, people say. It’s killing our schedule and budget, they conclude. So they work to erode those processes until another adverse event happens, albeit of a slightly different flavor. Rinse and repeat.
Structure from motion gets bad results from textureless solids where the edges might be a moving shadow.
The view diagrams on https://www.tesla.com/autopilot seem to indicate that two cameras would have had the columns in view. But if you're right, there's no good reason for it to be the case and perfectly demonstrates gross overreliance on single frame ML.
Yes. Pause the video and look at the car's screen. There's no indication on screen of the columns. A car on the other side of the row of columns is recognized, but not the columns. It's clear that Tesla has a special-purpose recognizer for "car".
The columns are a solid obstacle almost impinging into the road, one that doesn't look like a car. That's the standard Tesla fail. Most high-speed Tesla autopilot collisions have involved something that sticks out into a lane - fire truck, street sweeper, construction barrier - but didn't look like the rear end of a car.
As I've been saying for years now, the first job is to determine that the road ahead is flat enough to drive on. Then decide where you want to drive. I did the DARPA Grand Challenge 16 years ago, which was off-road, so that was the first problem to solve. Tesla has lane-following and smart cruise control, like other automakers, to which they've added some hacks to create the illusion of self-driving. But they just don't have the "verify road ahead is flat" technology.
Waymo does, and gets into far less trouble.
There's no fundamental reason LIDAR units have to be expensive. There are several approaches to flash/solid state LIDAR which could become cheap. They require custom ICs with unusual processes (InGaAs) that are very expensive when you make 10 of them, and not so bad at quantity 10,000,000. The mechanically scanned LIDAR units are now smaller and less expensive.
This matches my perception as well and it continues to blow my mind. Like it just seems like it would require full-on incompetence to use only specific pedestrian/car/sign/lane classifiers rather than splitting the world first into ground/obstacle by reconstructing geometry and then assigning relative velocity, acceleration, and curvature to coherent obstacle segments irrespective of their identification. But the videos always make it look like this is exactly what's happening. And worse it always appears to be happening on an instantaneous frame-by-frame basis with things flickering in and out of existence as if "thing there" and "nothing there" are somehow equally safe guesses.
It needs to be an input, not just a filter on output. A heuristic that doesn't account for the previously observed state is insufficient because prior existence information changes what gets detected. And it needs to lean heavily toward "if we saw something there before, the safest course of action is to assume that it's still there until strongly indicated otherwise".
> if we saw something there before, the safest course of action is to assume that it's still there until strongly indicated otherwise
Normally I'd agree with you, but driving a car is a bit of a special case since in may cases stomping on the brakes is way worse than doing nothing. Take the video of the tesla seeing spurious traffic lights[0] for instance, should it really do an emergency stop in the middle of traffic in that case?
All the machine learning can do is give you probabilities on what the state of the world around the car might be. The software needs to pick from those probabilities some state to act on in such a way to be the least likely to endanger anyone. There isn't a simple correct answer. all the variables need to be weighed. currently teslas seem to ignore some pretty important variables while driving.
Second, the cost of deciding that you're in the wrong lane is very different from the cost of deciding that there isn't an obstacle in front of you. The worst thing that happens when a navigator gets your position wrong is you miss a turn and spend a few extra minutes in the car. The worst thing that happens in the other case is someone dies instantly in a cloud of blood.
...and driving straight into barriers.
> the system doesn't know which input to trust
Well, clearly their camera network is untrustworthy, so...
However, it's good to see other automakers like Volvo going the LIDAR route on their 2022 electric models and are offering Level3 autonomous driving.
Also, Elon has gone on Twitter rants several times before against LIDAR. That horse has already left the stable and it ain't coming back.
Well that's just too bad.
While I'm sure it would be inconvenient to their business plan, it remains the case that they're charging ten grand for that software patch, and have no possible argument relating to margin.
https://thenextavenue.com/2020/06/10/teslas-model-3-profit-m...
Edit: nitrogen I can’t comment directly on yours, so I’ll ass it here. That’s a good goal to have and they tried that for well over two years. They found too much noise and confusion. If Radar says one thing and Vision says another over and over who do you trust? They decided they could train their NN with just Vision. They use LIDAR on test cars to confirm their Vision can recognize distances correctly. In this case I believe cutting out a sensor instead of trying to neuter it is the right decision.
If he contradicted his boss in public, Musk would just fire him and hire someone who will repeat the party line. I won't believe anything he says about lidar until he's no longer at Tesla, or until they have a car that's not trying to ram concrete pillars.
More sensors is better. They just have not yet learned how to merge data. Example is drone autopilot, that have IMU and GPS and uses Kalman filtering.
Neither? If you don't have a tiebreaker you disengage and let the human decide. We solved this problem in systems long ago. You either fail open or fail closed depending on the system. In this case it seems like you'd always fail to the safest option which is alert the human.
this too, means tesla is a blind man going by memory, not an autonomous driver.
The people writing the code, the people designing the cars, the people allowing the testing, etc etc.
What we're seeing in these videos is basically unusable and will never be cleared to drive by itself on real roads.
It's just the "the last 10% takes 90% of the time" adage but applied to a situation where you can kill the occupant of the car and/or the people outside it. And that last 10% will never be good enough in a general way.
No, i don't think this is really more than maybe 30% of the journey to FSD - and, most likely, according to what has been shown this time (and the last times) it'll never get there.
Legislators are slow and ignorant about technology. There's also perception that everyone who dies on a Tesla accident is a tech bro who defeated software safeguards (e.g., chose to ride on the back seat in full self-driving mode).
If automotive history is any guide, scrutiny and regulation only appear after a lot of people get killed. It's not obvious what the threshold will be, with Elon Musk being louder than anyone else in this debate and early adopters being comfortable experimenting with the lives of others.
I think this is pretty well understood to be the case. Level 5 FSD is magnitudes harder.
With that being said, even if it comes out of beta. I would likely only use it for extended highway travel in clear conditions.
I watched (and fast-forwarded) through a few of the original, full-length videos from which these clips were taken. The full-length videos show long drives (in some cases, hours long) almost entirely without human intervention, under conditions explicitly meant to be difficult for self-driving vehicles.
One thing I really liked seeing in the full-length videos is that FSD 9 is much better than previous efforts at requesting human intervention in advance, with plenty of time to react, when the software is confused by upcoming road situations. The handpicked clips are exceptions.
For BETA software, FSD 9 is doing remarkably well, in my view. I mean, it's clearly NOT yet ready for wide release, but it's much closer than all previous versions of Tesla FSD I've seen before, and acceptable for a closed-to-the-public Beta program.
> For BETA software, FSD 9 is doing remarkably well
If this was a React website, that'd be great. But it's a production $40,000 multi ton automobile.
Are you OK with that or do you think we should attempt to fix that with software? If you do think you should attempt to fix that, do you understand that software engineering is a iterative process? It gets safer overtime.
That's a straw man argument. Mandatory breathalyzer tests before every engine start and yearly medical and psychological check-ups would solve a lot of these problems as well.
> It gets safer overtime.
There is no evidence of this happening yet. It's still trying to drive people into concrete pillars. Remember that you only need to drive into a pillar once to die. Passing it 10,000 times before doesn't help when an update suddenly decides that the pillar is now a lane.
I'd be totally fine with Tesla testing this with trained drivers in suitable areas but giving this into the hands of amateurs is pure "move fast and break things", with "break" meaning ending lives.
The worst thing is: all of these videos feature sunny weather. How bad would it be in the rain? Ever tried to take picture in the dark with even light rain? Cameras are very bad at this.
These drivers are vetted and their usage is monitored, many are employees. In the long run automated driving will save many more lives than it takes. Just like automated elevators.
https://www.npr.org/2015/07/31/427990392/remembering-when-dr....
This does not seem like an important nit to pick, so I'm wondering why you're picking it. The number of people killed or injured because a driver had a heart attack at the wheel is vanishingly small compared to drunk or distracted driving.
So let's reframe your question as you should have asked it:
"If someone is driving drunk or distracted we should take away their license?"
Yes. Absolutely.
We don't do that today and we are still allowing people to drive on the road today.
That in itself doesn't constitute evidence that it isn't getting safer. After all humans occasionally drive into concrete pillars too.
What matters is that the distance traveled without incident goes up in a super-linear fashion over time. It would be nice if someone had hard numbers on that.
But that's besides the point. I was arguing about the trend line. It may seem silly now, but if the error rate is roughly a straight line on a logarithmic graph then we'll get there soon enough. If it isn't then tesla at very least would have to admit they have a problem and explain how they'll get out of it.
> There is no evidence of this happening yet.
With aircraft any crash or near-crash is investigated and then the software and procedures for all similar aircraft are updated. Over the decades this has resulted in a huge reduction in the rate of accidents. Isn't it likely that the same thing will happen with self-driving cars?
Based on the full-length videos, I'd say it's more like 99.9% or even 99.99% for FSD 9.
Human drivers don't steer towards concrete pillars every time they drive down the street.
The new software seen here hasn’t been released so there’s no data yet. We cannot possibly make an assessment based on a few hand-picked videos - a quick YouTube search will give you thousands of human driver examples that are even more dumb.
> In the 1st quarter [2021], we registered one accident for every 4.19 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot but with our active safety features, we registered one accident for every 2.05 million miles driven. For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles.
Not really a great metric. We should be looking at disengagements per MM.
The part of the discussion that concerns their marketing is uninteresting and overdone; someone else can discuss that part.
Their autopilot stat is also misleading because AP will most likely be engaged only in ideal weather conditions on divided roads.
Mostly the Tesla numbers are comparing accident rates of just Teslas. You can ignore average accident rate across all vehicles if you like.
If would be truly great if the NHTSA could publish accident rates broken down is various ways like by road type and weather conditions. I’ve never been able to find this data.
The point being that the failure rates society accepts for cars are actually pretty amazingly low and human drivers are surprisingly good. These things are measured with 6-7 9s as a non-negotiable minimum, so positing hypotheticals at the 99-99.99% is highly non-representative of what is actually acceptable.
Anybody discussing things at that level is making a pretty fundamental category error and any conclusion that anyone might draw from those thought experiments are largely invalid due to the multiple order of magnitude difference between such a thought experiment and reality.
Given the telemetry the car is equipped with they absolutely have the data to control for some of these factors yet they make no attempt to do so. Why is that? My take is that after doing so autopilot doesn't look so good.
You assumed numbers are time. Now assume numbers are manoeuvres.
Also, the fact that they're distributing safety critical software to the public as a 'beta' is just insanity. How many more people need to die as a result of Autopilot?
I agree. Everyone agrees. That's why FSD Beta 9 is closed to the public. My understanding is that only a few thousand approved drivers can use it.
> Also, the fact that they're distributing safety critical software to the public as a 'beta' is just insanity. How many more people need to die as a result of Autopilot?
FSD 9 isn't being "distributed to the public." It's a closed Beta. Please don't attack a straw-man.
A closed beta needs to happen on private roads. If it happens on public roads it is not a closed beta.
Some of the videos show drivers having to agree to take full responsibility for all driving.
Also, Elon keeps promising "the button" which would allow any Tesla FSD "owner" to get the software.
A number of people shelled out $10k for this software that they do not yet have. So there is a real possibility that it may soon be made available to everyone.
The videos in the Twitter feed are nothing like that. The car makes potentially catastrophic blunders, like driving straight into a concrete pylon, with 100% confidence.
As a pedestrian and someone who shares the road with drivers who use FSD, I don't get to opt out of this "closed beta", and I certainly wouldn't care about how many hours of quality content it has on YouTube if it caused a car to hit me or drove me off the road.
But IMHO it's not full self driving if it requests the driver to take over even once.
If there's an insane storm or something then it's ok for FSD to know it should disable and then you have to drive 100% control. The middle ground is more like assisted driving which doesn't seem safe according to most HN comments.
Also, this beta isn't "closed-to-the-public". The "public" is an active and unwilling participant in it.
The 99% of 'good' doesn't matter if you keep driving into giant barriers.
Cherry picking is completely fine because with the limited number of beta users, even a few incidents is enough to show that FSD isn't nearly ready for level 5.
And it's pretty easy to make a car that can self drive on simple roads without a tonne of traffic, so being able to do that without intervention isn't much of an accomplishment.
IMHO this puts everybody and their children at risk so Tesla can beta test earlier, but I would love to corrected.
What would this entail? Perhaps some sort of "license" which allows the user to operator a motor vehicle?
Novel skill requirements include: interpreting FSD UI, anticipating common errors, recognizing intent behind FSD actions, & remaining vigilant even during periods of autonomous success.
The opposite - where the automation handles the common case, with the human watching, simply does not work. It's not worked in aviation, it's not worked in industrial systems, it's not worked in self driving cars. This isn't a novel bit of research, this is well understood.
"I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work.
Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety.
I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone.
At least with an unreliable system that Tesla demonstrated here.
And we still get Boeing-style @&$#-ups at the UX interface.
"You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important.
"In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.
Solving for that, even a system that was extremely conservative would be useful, as you are completely freed up when it is operating, even if that is only 20% of the time.
The horse could go crazy in scary situations endangering others but you could train that out and make a good war horse.
Basic stuff like following the road around a bend was something the horse could do but you could not tell it to autonomously go across town to pick someone up.
People were okay with horses and knew what to expect. Tesla need to rebrand autopilot to 'horse mode' and reset people's expectations of FSD.
FSD is a complete alien that might try to kill itself for the most trivial and inscrutable reason like the sensor noise excited the neural network in a certain way.
https://www.youtube.com/watch?v=2ft9OXQ03qE (you can stop after the first hit because it repeats in loop) (I chose the less shocked one, some of them are terrible).
Do you have thoughts on why this is? Is it because, almost by definition, the remaining 0.1% are the toughest problems so the hardest to catch/solve? Or is it a human-factors issue where people get complacent when they know they can count on the machine 99.9% of the time and they lose the discipline to be vigilant enough to catch the remaining problems?
Basically if you grow up with a self driving car and never learn how to handle small fuck ups, how can you ever be expected to recover from a big fuckup? (Bad sensor reading, bad weather, shredded tire, etc)
Anticipating common errors and remaining vigilant while driving is a requirement for all driving, not when dealing with FSD.
If you are capable of driving a car safely to begin with, then you're capable of recognizing then the car is going to do something wrong and turning the wheel and/or hitting the brakes.
I wouldn't personally trust a FSD beta, but only if it is properly supervised, I think it could be tested safely. But, there's the problem -- how can you really properly supervise this? How many drivers are going to try this and just let the car drive w/o paying attention? Or, using it under controlled conditions, but then giving it too much trust in more adverse conditions?
Unless the user is able to understand and anticipate common reactions, the entire system is unsafe at >25 mph or so.
Or, to put it another way, I'd love to see the statistics on how many drivers can react quickly enough to avert an unexpected turn into an exit divider at 60 mph, after a number of days of interruption free FSD.
Being a cop on duty often exempts you from distracted driving laws.
I'm not sure how they can do stuff on a laptop while driving safely but I assume the state knows best. /s
A motorcycle is similar to an automobile, but has different failure modes, and needs a special license. A >5 tonne truck is very similar to an automobile, but has different failure modes, and needs a special license. An automobile that usually drives itself, but sometimes tries to crash itself has different failure modes from an automobile that does not try to drive itself.
It fails in ways that are totally different from how humans typically fail at driving.
Student drivers have the ability to comprehend and understand the 3D world around them - it's just a matter of learning the proper control manipulations to make the car do what you want, and then how to interact with other traffic in safe and expected ways.
Tesla's FSD system seems to fail at basic 3D world comprehension and reading, often enough.
Average folks beta testing Tesla's software for them are woefully under-prepared to safely mitigate the risks these vehicles pose. In several cases they've meaninglessly died so that Tesla could learn a lesson everyone else saw coming months in advance.
The guy with the white ball cap repeatedly attempted a left turn across two lanes of fast-moving traffic, with a drone providing overhead views. He seemed smart, aware, and even a bit conservative in intervening. Nonetheless, I couldn't help but thinking that none of the oncoming vehicles consented to the experiments racking up YouTube views. If he doesn't jump on the brakes at the right time, he potentially causes a head-on collision with a good chance of fatalities.
Yes, I do agree that beta drivers should get extra training. However, I'm not sure I agree with the premise of beta testing an aluminum rocket on wheels on public roads in the first place.
A March 2021 tweet indicated that beta access will be as simple as pressing a "Download Beta" button in the car. A subsequent tweet said that the program had 2,000 owners, some of whom have had their access revoked for not paying enough attention to the road.
It's even crazier that we allow that. The cars that student drivers learn in typically do not have dual control so it's quite difficult to intervene. Every highschool has a story about how a student slammed the accelerator instead of the brake.
Regulating self driving is a good idea but it's a tall order to cast beta software as the menace when there are more idiots driving on the roads at this moment than there will ever be Tesla products.
Besides that, FSD being less dangerous than drunk driving is a terrible benchmark.
Tesla FSD driving assist has never killed someone not in the car.
Considering that rampant safetyism is par for the course here, I’m hardly convinced of the “he could’ve killed someone” school of thought.
Thanks for the info!
[1] https://www.nytimes.com/2018/03/19/technology/uber-driverles...
When you look at the dashboard visualizations of the cars‘ surroundings, the model that is built up looks quirky and inconsistent. Other cars flicker into view for one frame and disappear again; lane markings come and go. I saw a video where a car in front of the Tesla indicated, and the traffic light in the visualization (wrongly) started switching from red to green and back, in sync with the indicator blinking.
How could a car behave correctly as long as its surroundings model is so flawed? As long as the dashboard viz isn’t a perfect mirror of what’s outside, this simply cannot work.
It is ridiculous, that the model doesn’t keep track of objects, and assume they continue with current velocity when they become obscured. It seems like a relatively simple thing to add depending on how they represent objects. You could even determine when the objects are obscured by other objects.
In 8.x videos I noticed cars shifting and rotating a lot over fractions of a second, so it seemed like they needed a Kalman filter for objects and roads.
Objects in 9.0 look more stable, but I still see lanes, curbs, and entire intersections shifting noticeably from frame to frame. So if they added time (multiple frames) to the model, it is still not working that well.
It's like the ML system needs its inferences constrained by rules like "objects don't teleport" or "acceleration is never infinite."
It's not 2 weeks or whatever unrealistic timeline away from being done, as Elon has claimed for ever. 2 perhaps years if we're lucky, but given human and driving complexity probably way more before even the whole of the USA is reliably supported beyond L2.
FSD would be equivalent to the Mexican border wall, I guess?
I think it depends what they actually are trying to accomplish. This is Beta for a glorified cruise control overhaul; not a beta for promised RoboTaxi.
Musk/Tesla tend to talk about RoboTaxi then slip seemlessly into/out of 'but today we have low engagement cruise control!'.
Fair bit of hucksterism.
Good point. "Full Self Driving" in my mind paints a picture beyond "a better cruise control". But maybe they meant that and just named it wrong.
> Full Self-Driving Capability
>
> All new Tesla cars have the hardware needed in the future for full self-driving in almost all circumstances. The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat.
Even if "on aggregate" FSD performs safer (by some metric) than humans, as long as FSD continues to fail in a way that would have been easily preventable had a human been at the wheel, FSD will not be accepted into society.
Except for when the radar doesn’t see an obstacle in front of you, eg because the car in front of you just changed lanes. That needs to be looked out for.
There was nothing spectacular about those failures, I would say because the driver was attentive and caught/corrected the car. That's not to say some of these fails could not have ended in catastrophe, but to call them spectacular is quite the exaggeration.
One of those "spectacular fails" was displaying two stop signs in the UI on top of each other while properly treating it as one stop.
Using hyperbole like this only makes people ignore or dismiss your otherwise valid point.
It does not matter that any autonomous driving tech is safer than human drivers. They MUST be perfect for the general public to accept them. The only accidents they'd be allowed to get into are ones that are beyond their control.
Algorithmic accidents, no matter how rare they are, won't be tolerated by the general public. Nobody will accept a self driving car running over a cat or rear ending a bus even if regular humans do that all day long.
The expectation for self driving cars is a perfectly attentive driver making correct decisions. Because, that's what you theoretically have. The computer's mind doesn't "wander" and it can't be distracted. There's no excuse for it to drive worse than the best human driver.
So you're right, the only bar to reach is perfection (which is impossible), because algorithmic errors have biases that will likely deviate from human biases.
That said, there are going to be a lot of dead small animals due to autonomous vehicles. I'd hope that whoever develops the system has some good training data to stop it from hitting children.
The issue will be that it's going to be real hard to make a system that can tell the difference between a plastic bag and a poodle.
This is already true today of human drivers because of the tall SUVs that are so popular. Do you think matching biases will be acceptable?
I'd say, absolutely not. The only way we'd accept that is if the kid darted out before the vehicle could slow down, and even then we'd expect super human braking to (hopefully) avoid serious injury.
Also, self driving cars have and advantage that they can put cameras in places typical drivers eyes aren't. They should be able to see a lot more than you can from the driver's seat.
First, the vast majority of pedestrian deaths are adults. In 2018, a total of 206 age 15 or younger were killed by cars. Compare that to 5,965 killed who were age 16 or older.[1] Both in absolute numbers and relative to population, children are far less likely to be run over and killed than adults.
Second, while light trucks (vans, SUVs, & pickups) are 1.45x more deadly to pedestrians than cars, buses are far more dangerous than either. Motorcycles (which have excellent visibility) are particularly deadly to child pedestrians. From United States pedestrian fatality rates by vehicle type[2]:
> Compared with cars, the RR of killing a pedestrian per vehicle mile was 7.97 (95% CI 6.33 to 10.04) for buses; 1.93 (95% CI 1.30 to 2.86) for motorcycles; 1.45 (95% CI 1.37 to 1.55) for light trucks, and 0.96 (95% CI 0.79 to 1.18) for heavy trucks. Compared with cars, buses were 11.85 times (95% CI 6.07 to 23.12) and motorcycles were 3.77 times (95% CI 1.40 to 10.20) more likely per mile to kill children 0–14 years old. Buses were 16.70 times (95% CI 7.30 to 38.19) more likely to kill adults age 85 or older than were cars. The risk of killing a pedestrian per vehicle mile traveled in an urban area was 1.57 times (95% CI 1.47 to 1.67) the risk in a rural area.
All else equal, being hit by a larger vehicle does increase the risk of severe injury or death, but all else isn't equal. Larger vehicles tend to be more visible, louder, and slower than their smaller counterparts. Different types of vehicles are driven in different environments with different propensities for mingling with pedestrians. If vehicle mass and blind spots were the main factors in pedestrian deaths, we should have seen deaths skyrocket over the past 40 years (as cars got bigger and bulkier for greater passenger safety). Instead we saw pedestrian deaths decrease.
1. https://docs.google.com/spreadsheets/d/e/2PACX-1vRqGqodKkWkS...
No they don't, its far from perfect right now yet its available and you can use it right now provided you have money to buy it.
AFAIK, the closest there is WAYMO, but they are geo-fenced. There's also some fixed route low speed buses out there. However, there's no self driving you can purchase which allows you to, for example, nap while the car is driving.
So are you saying people accept it in the far from perfect driver stage now but won't accept it when it become so much improved "no driver at the wheel" stage ?
I'm saying that the "no driver at the wheel" stage must be flawless. I've seen some claims that it's "Good enough if it's x times better than a human driver" or "can drive n number of miles without an accident". However, both of those are not the metrics to measure.
A "no driver at the wheel" package is good enough when it doesn't run over the neighbor's cat or Timmy on the street. Humans today make that mistake all the time, but that's not a mistake a driverless car can make and get away with. Consider, for example, Uber's killed pedestrian. Sure, it was a tough scenario and the driver wasn't paying attention. But the fact of the matter is everyone had the expectation that the Uber car would not hit that pedestrian even if they were really hard to see.
Until that happens, nobody will accept "no driver at the wheel" cars... except maybe in tightly controlled routes/geofences. Otherwise, SDC will require a driver to be attentive at the wheel at all times. That is, telsa's FSD is a very long way away from being able to hit that "telsa taxi service" that elon has pitched. I doubt it can make it with the current sensor set (not enough redundancy).
I too expect and accept that some accident caused by driverless car that kills kid/animal (that human wouldn't do) will happen (if not already happen). To expect otherwise will be unrealistic.
Of course this is all theoretical. If we had solid evidence that it performs better than humans in some scenarios but worse in others then we could save even more lives by only allowing it to run in those cases where it does and only do shadow-mode piloting in the others (or those who opt into lab rat mode). Enabling it by default only makes sense if we do know that it performs better on average and we do not know when it does.
But you can also be a pedestrian or passenger. Do you not want everyone else to be less likely to kill you?
Also, should you really trust your own estimate of your driving safety?
> McCormick, Walkey and Green (1986) found similar results in their study, asking 178 participants to evaluate their position on eight different dimensions of driving skills (examples include the "dangerous–safe" dimension and the "considerate–inconsiderate" dimension). Only a small minority rated themselves as below the median, and when all eight dimensions were considered together it was found that almost 80% of participants had evaluated themselves as being an above-average driver.[30]
Statistically autopilot gets in fewer accidents than the average human driver. And if you believe you are an above average driver, why do you believe you would be unable to intervene if needed?
It's far from clear, in practice, if they're actually doing this. If they have, it would have to be fairly recent, because the list of "Oh, yeah, Autopilot always screws up at this highway split..." is more or less endless.
GM's Supercruise relies on fairly solid maps of the areas of operation (mostly limited access highways), so it has an understanding of "what should be there" it can work off and it seems to handle the mapped areas competently.
But the problem here is that the learning requires humans taking over, and telling the automation, "No, you're wrong." And then being able to distill that into something useful for other cars - because the human who took over may not have really done the correct thing, just the "Oh FFS, this car is being stupid, no, THAT lane!" thing.
And FSD doesn't get that kind of feedback anyway. It's only with a human in the loop that you can learn from how humans handle stuff.
It's a QA department. If there is a failure hot spot, then take a bunch of known "good" QA drivers through that area. Assign strong weight to their performance/route/etc.
It's interesting reading through all this, I can see a review procedure checklist:
- show me how you take hotspot information into account
- show me how your QA department helps direct the software
- show me how your software handles the following known scenarios (kids, deer, trains, deer weather)
- show me how you communicate uncertainty and requests for help from the driver
- show me if there is plans for a central monitoring/manual takeover service
- show me how it handles construction
Also, construction absolutely needs to evolve convergently with self driving. Cones are... ok, but some of those people leaning on shovels need to update systems with information on what is being worked on and what is cordoned off.
No. If the car cannot handle random obstructions and diversions without external data, it cannot be allowed on the road.
Construction is often enough planned ahead of time, but crashes happen, will continue to happen, and if a SDC can't handle being routed around a crash scene without someone having updated some cloud somewhere, it shouldn't be allowed to drive.
First responders need to deal with the accident, not be focused on uploading details of the routing around the crash before they can trust other cars to not blindly drive into the crash scene because it was stationary and not on a map.
And if you can handle that on-car, which I consider a hard requirement, then why not simply use that logic for all the cases involving detours and lane closures?
Doing so ahead of time for planned construction is not a big ask.
"And if you can handle that on-car, which I consider a hard requirement, then why not simply use that logic for all the cases involving detours and lane closures?"
You're making the same mistake Musk made when insisting that the car be able to navigate regardless of location or connectivity. Ignoring the ability of networking/radio broadcast/internet databases to provide vastly more deep information pools is a big mistake.
I guess I sort of assumed that Tesla would do three things:
- Record the IRL decisions of 100k drivers.
- Running FSD in the background, compare FSD decisions with those IRL decisions. Forward all deltas to the mothership for further analysis.
- Some kind of boid, herd behavior. If all the other cars drive around the monorail column, or going one direction on a one way roadway, to follow suit.
To your point, there should probably also be some sort of geolocated decision memory. eg When at this intersection, remember that X times we ultimately did this action.
The FSD could then infer that no other cars passed thru meridians, planters, and columns. It could infer that only busses travel in restricted lanes. It could infer that all traffic on a one-way road goes one way.
And if FSD remembered its own decision every prior time, it could reconfirm its current decision.
In other words, it could learn from every other vehicle and its own history.
Routes people drive frequently are much more optimized: knowledge of specific road conditions like potholes, undulations, sight lines, etc.
I would like to have centrally curated AI programs for routes rather than a solve-everything adhoc program like Tesla is doing.
However, the adhoc/memoryless model will still work ok on highway miles I would guess.
What I really want is extremely safe highway driving more than automated a trip to Taco Bell.
I personally think Tesla is doing ...ok. The beta 9 is marginally better than the beta 8 from the youtubes I've seen. Neither are ready for primetime, but both are impressive technical demonstrations.
If they did a full-from-scratch about three or four years ago then this is frankly pretty amazing.
Of course with Tesla you have the fanboys (he is the technogod of the future!) and the rabid haters (someone equated him with Donald Trump, please).
A basic uncertainty lookup map would probably be a good thing. How many tesla drivers took control in this area/section? What is the reported certainties by the software for this area/section?
It's all a black box, google's geofencing, Tesla, once-upon-a-time Uber, GM supercruise, etc.
A twitter account listing failures is meaningless without the grand scheme of statistics and success rates. A Twitter account of human failures would be even scarier.
Look, Radar, Lidar, and Maps aren't perfect. A map can be wrong, Radar can have false echos, Lidar can have issues in bad weather, etc. But if you're depending on visual recognition of one-way signs, having a potentially out-of-date backup map to check your prediction again, and err on the side of caution, is better than having no Map at all.
Ditto for radar and lidar. Radar enabled cars have collided with semis, remember, because they're aimed low (underneath the semi). But in the case of these planters, or road cones, or monorail poles, the radar is going to be a strong signal to the rest of the system that it isn't seeing something.
There's no way FSD should be allowed on public roads, it's not even CLOSE to ready. The other car companies are behaving far more responsibility: full sensor suite, multi-sensor fusion, comparison with detailed road maps, and limiting operation to safe areas, and expanding as safety is proven.
Who the hell cares if it's not FSD driving if its so dangerous? I'll take 75% FSD that only operates on highways and long commutes or road trips, over 100% FSD that has so many disengagements and "woah" moments as to be unusable.
Waymo self-driving is a relatively large scale. It also has blunders but they appear to be far less concerning than Tesla's.
I mean, just look at the visual representation of what the FSD is showing. It has no temporal stability. Solid objects are constantly appearing and disappearing. Often the trajectory line periodically switches from a sane path to some radically wrong path.
The car then often takes a HARD accelerated left or right, like it is dodging something, which often isn't there, only to threaten another collision.
It's like a Trolley Problem where one of the tracks is empty, and you divert the trolley into a group of kids.
The "aggressive" nature of its fails is super concerning. My son is 16 and learning to drive right now. When he makes a mistake, he doesn't aggressively accelerate and make wild turns, he hits the brakes. In most cases, in residential traffic (25-30mph), someone braking will annoy other drivers, but they will be able to stop and avoid collision. At many city driving speeds, you should not have to make sudden, super aggressive turns into the other lanes. Just safety come to a stop, even if you annoy people.
I see Musk's attitude here as endangering the public. Cars aren't rockets launched at far away landing strips by expert flight controllers. Even these beta testers can fuck up, as numerous people have already died using FSD, because some dipshits like to watch DVDs or sit in the back seats to show off to friends on TikTok, dangering not just themselves, but the public.
The beta should be closed IMHO to Tesla employees only and professionally hired drivers. I mean, why can't Tesla just do what other companies do, hire like 1000 drivers to go drive FSD around 100 American cities for months at a time collecting data?
If you watch the full videos and not a mash up, you’ll see most drivers drive long routes without any problems with the software. It’s the edge cases you’re seeing and yes you can make a compilation with them and make it look horrible.
Honest question, do you think a Tesla that doesn’t recognize whats in front of it to 100% should stop in the road? I saw this recently with Waymo and it stayed blocking traffic until a Waymo tech arrived then the car tried to drive away from the tech. This doesn’t seem like the better answer to me, but I don’t have one.
As far as comparing it to a teenage driver, they are the most dangerous drivers on the road and Tesla’s happen to be the safest rated cars in accidents. Interesting combo.
Can you please link an article about FSD and deaths? There have been a few on auto pilot when the drivers are not paying attention but I’m not aware of a single FSD death.
Why can’t other companies make self driving software/hardware that a normal person can afford? All other options are insanely expensive and out of reach of anyone but millionaires.
Answer? Yes. A fully stopped car is a better option than a car that plows into a wall, barriers, or pedestrian.
> Why can’t other companies make self driving software/hardware that a normal person can afford? All other options are insanely expensive and out of reach of anyone but millionaires.
Not sure what you're talking about by "other options". Every other large company I know of is either 1) pursuing self-driving for ride-hailing services, or fleet operations or 2) still not shipping.
But Tesla's aren't exactly cheap, and Tesla cars aren't exactly in the affordable range ($20k) of the working man. A Model3 is $50k with FSD. $60-70k with the performance/long distance versions, and Model S/X well over $100k.
Perhaps you are talking about LIDAR costs and using the old $70,000 LIDAR system that Tesla originally put out FUD about. We all know that in mass manufacturing, the cost of LIDAR would fall, and it has, Velodyne has units that cost only $500 now.
A fully enabled array of consumer Velodyne units, plus radars, and cameras, is going to cost pretty much on par with what Tesla is charging.
Granted that the US has a lot of road-miles, but doing this work incrementally is the way to go. I have a feeling it will be faster to outfit the roads than to build reliable self-driving software. I will also grant that this is a "feeling" and I'm sure someone has done actual research :)
If this capability could be used to help hasten the move to electric vehicles, we could even get a climate change win in at the same time.
https://wgntv.com/news/when-its-this-cold-chicago-sets-its-t...
Weirdest thing that prevents trains from running tho is wet leaves on the tracks, of all things, it’s too slippery.
Self driving cars might well refuse to move in bad conditions (my Prius gets very upset when navigating my overgrown driveway for instance, sometimes slamming on the auto-brake to avoid the plant-pedestrians)
I think we'll be on the right track when city planners reclaim some of our current roads and make them available only to autonomous vehicles. They'll need their own garages, highways and streets to operate to fully realize it. For example, FSD vehicles can be moved to dedicated streets similar to how alleys work in many cities currently (where they're largely routes for maintenance and utilities -- removing them from normal commuter traffic).
I'd rather subsidize automated electric trolley lines in cities than subsidize exclusive roads for automated personal vehicles.
That doesn't mean I don't want any roads for private vehicles. Just that need not make up 99% of them. Even if we knocked that down to 75% of roads I think we'd see amazing positive impacts on every metric (except the private roads; those will continue to suffer from congestion until cities offer a better way to get people from A to B).
Not any more friendly than they are to humans. AI driving needs to handle the general case. Otherwise a road that has a lapse in autonomous-friendlyness could be catastrophic.
I could see some stretches of highways becoming automation friendly, but it wouldn't be anywhere near even a tenth of them.
Some may die, be disabled or may linger in vegetative state life-long, but it was their choice afterall and the it can argued that medicinal side-effects are very small cause of mortality and hence long-term the "public beta tests" will make drugs more effective and save more lives!
If you're asking whether that would be awesome if it lead to pharmaceutical innovations I think it would.
The pharma company could see results (self-driving car data) and figure out what caused the issues (details of the deaths in pharma, accidents in Tesla) and use that to make the next beta version.
I don't really see how that isn't an apt analogy.
It's called clinical trials for pharma the key point being it's opt in and doesn't affect anyone around the subject, unlike Tesla's autopilot beta.
Sure it does. It identifies cases where the drug may have unexpected side effects so either the chemistry, dosage, or expected risk factors can be refined.
To me autonomous vehicles are similar - the people who have access to them know they're not perfect, but they're willing to spend the money because they think it's better than the alternative
Why do Tesla drivers get the privilege to beta-test their cars on public road with everyone else around and potentially in their kill-zone?
Are you against Tesla owners in particular? I can hit and kill someone in any car with or without driver assist technologies. Are you worried that it's more likely that I will kill someone while using technology that is invented to make that less likely?
On the other hand they secured billions of $ for R&D into FSD and BEV design and brought EV production to scale. Their messaging is way to optimistic (borderline fraudulent) but they are doing better things with the money than the alternatives. Traditional automakers have shown to be completely impotent in this regard. Money is flowing into all kinds of wasteful bullshit (that Tesla is backing BTC makes them less trustworthy in my eyes)
If you offered me a car that drove itself, and statistically, it killed people at the same rate as humans, but let me not have to drive, I'd take that. Nobody everybody agrees.
Look at the American Gun debate as an example; carrying for self defense makes you drastically less safe by most metrics. But people prefer to have the modicum of self determinism over more consistent statistical safety.
You're probably referencing studies such as Investigating the Link Between Gun Possession and Gun Assault[1], which looked at whether people who were shot were carrying a gun at the time. It did not differentiate between criminals who were carrying illegally and law-abiding citizens who had concealed carry permits. The selection effect in such studies is clear: Many criminals who carry guns are involved in shootings. Also, many people carry guns because they are at a higher risk of being a victim of violent crime. I have yet to find a study that corrects for these confounders.
But the reverse is true. Humans receive driving bans and even criminal penalties for the sort of driving errors autonomous systems make without penalty.
That's not how it even works though. When people crash or drive drunk there is a system to hand out consequences. What do you do when a Tesla drives into a monorail pole and causes millions in damage if the structural integrity is compromised?
>statistically, it killed people at the same rate as humans
You can't tell me a car that's at best an average not-murderer is a good sell.
There are also so many ways to self-determine your risk while driving or traveling in general. A clear example is seat belts. Less than 10% of people in the US don't wear seat belts but a full 47% of the people that died in car accidents were not wearing one. [1]
I personally enjoy playing with the progress of autopilot over the years and I'd be sad to see it more restricted.
I understand that it can be unsafe if left unsupervised but in reality I've never met someone who drives like that.
If this has been the case, evidence here is certainly lacking. Turning down a street one presumably hasn't seen a car drive down in this direction? Failing basic stationary obstacle avoidance because they're in the center of the road? Screwing up basic turn lane behavior?
Either they aren't taking the "years of learning" data into account, or, worse, they are - and even with all that correction, it's still this bad.
As far as I know they've been using the fleet to collect raw camera data for their training set but I don't think they've used the fleet to learn driving behaviour.
> repeatedly tries to drive into the monorail pillars
Choose one and only one
As someone in another thread said:
Driving with Tesla FSD is like being a drive instructor for a narcoleptic student driver who is currently tripping balls.
I don't want to share a road with these things. I try to stay as far away from every tesla on the road as I can.
This is just nonsense. These videos show them at a disengagement every couple miles. Waymo was at that level 10 years ago and is 10000s of times better now.
[1] https://arstechnica.com/cars/2021/05/why-hasnt-waymo-expande... [2] https://www.bbc.com/news/technology-57144755
Waymo doesn't require frequent human intervention to prevent life threatening situations.
Did you know 38,000 people die in America each year from auto accidents and another 4,400,000 are injured yearly [1]. I would like to see a decrease in unnecessary deaths and accidents on the road. Self driving cars are the future one way or another because they will be safer than humans. Tesla FSD is in beta, in the few hands of what Tesla has deemed as safe drivers based on their driving data. There have been no accidents on FSD and no deaths either. Autopilot is not the same as FSD.
If you care about reducing automotive deaths then you should agree that reckless "full self driving" functionality which is anything but is a concern: behavior like this will create public and regulatory pushback which ultimately delays the availability of safety improving technology.
The government allows Tesla to conduct a randomised control trial: When you get in a Tesla you enter your destination, then there's a 50% chance of having to drive the car manually (the 'control') and 50% chance of having the car drive you (the 'treatment'). After 10 million trips, if the treatment group had fewer accidents than the control group, then Tesla is allowed to roll out self driving to everyone. But on the other hand, if treatment has more accidents than the control group then Tesla must pay, say, $10 million for each excess fatality.
Yes, I get it, humans use 2 eyes only, but sensor fusion of complementary technologies is so powerful. When lives are on the line, having more than one type of sensor involved is just prudent engineering.
Getting city driving working just seems so impossibly hard, there is no way you are going to be able to use ML to determine road rules since they vary by jurisdiction and over time, and there are so many edge cases you won't have enough data for. But how do you write a program that can read a street sign, interpret it's meaning, and act upon it safely?
Even if Tesla just got autopilot to the point where it could be level 3, where drivers don't have to pay attention, that would be a massive improvement, and so much easier than FSD.
Maybe they painted themselves into a corner though with all Elon's blatant lies about FSD, can't stop development without a lawsuit. They seem kind of screwed anyways, looks like they are running into the limits of the current camera positioning and onboard processing power, might need to do another iteration of replacing every FSD owners hardware.
However today, isn't Tesla profitable now? They already have created a Veblen good, what benefit does Tesla get out of a high risk feature?
Sure, test autonomous driving off the roads or on dedicated test environments, but do not permit this on public roads, i do not wanna be part of any beta test.
It's actually pretty simple: have FSD do a regular driving test. If it can pass that it's good to go, if not it fails the test and will not be allowed to control a vehicle.
> 4th: Tesla doesn't recognize a one-way street and the one-way sign in the street, and it drives towards the wrong way
If the car is in Boston this is OK.
What about the multiple instances where FSD drives the car forcibly into a firetruck?
Because the alternative right now is... this abomination that doesn't even grasp what a concrete pillar is?
The fact that you're saying he put hundreds of lives at risk when a hundred is a score that a terrorist with a semi truck would be proud of and is for all practical purposes unattainable with a light vehicle does seem to point in that direction.
I'm not condoning his behavior but 1) people are jerks, what do you expect and 2) you are getting way to bent out of shape over it.
> Now we have email, and in the future we will have (actual, real) full self driving.
And yet people pick up the phone for the urgent stuff and send snail mail for the super important stuff (though the latter is changing with online forms moving into more and more areas). That says something about the true nature of technological progress. We come up with new better stuff but often old stuff keeps some niches for itself.
I think people will naturally be much more critical of a car than human drivers even if "overall" they are statistically safer.