A former pilot on why autonomous vehicles are so risky
spectrum.ieee.org
spectrum.ieee.org
I think that these self driving cars need some kind of feedback system for pedestrians and cyclists. Even an emoji face that shows facial expressions would help. When I see a real driver at an intersection, I feel like we come to some agreement about what is going to happen next based on our facial expressions. Or I can at least guess what may happen and take an action. With the self driving car there is 0 feed back. It's just an empty car that may or may not run me over.
Additionally, I also almost got run over by a dude head down driving in the Mission the other day, typing on his phone with both hands as his he just felt comfy having his Tesla do all the work.
At the moment all of this feels so sketchy. I hate my life being used for beta testing software lol.
At one point I saw a proposal to put cartoon eyes on self driving cars that could turn and “look” at you to achieve this effect.
Actually seems like a good idea to me!
(I'm not trying to imply the agent may fool you into crossing so it can mow you down, the case of "AI incompetence" so to speak is more concerning to me then "AI malice".)
https://www.youtube.com/watch?v=1Za8BtLgKv8
Maybe a carton of eggs would achieve the same message? Don't want to do what those kids in Aurora did.
As someone who struggles with nonverbal communication sometimes, I frequently feel this way about all vehicles both as a pedestrian and also as a driver myself.
To say nothing of other safety critical industries.
We know what works ok. We know what’s dangerous. We know what’s lethal. There are lessons to look at.
And yet Tesla and a lot of people seem content to re-learn the same lessons through blood over and over again. Like this is 100% incomparable to anything that’s ever happened before.
Tech is so powerful now, it controls narratives, influences politics, huge revenues.
I feel tech is beyond control at the moment.
But she clearly shows how we are currently in the most dangerous phase “car drives except handing off to human drivers in a crisis”.
“ The policy should be that either the computer is driving or you are driving. And by driving I mean steering—people do fine with regular cruise control. The act of keeping your hands on the wheel and guiding the car’s lateral motion is enough to keep your brain engaged. So, no L3 [full self-driving, but the driver must be ready to take the wheel], which is too confusing, and no hands-free L2 [partial self-driving]. I am not against the passing of control per se, but there should just be two modes of operation, with crystal clear feedback about which mode you are in.”
FSD has been 5 years away for… 15 years now?
this was only Elon-time hallucinations that made for great marketing
you won't get to Mars on Elon-time either, but that doesn't mean you won't get there
Doesn't mean you will either.
FSD is going to be enabled on HW2 anyday now. Oh look at that HW4 just rolled out.
the past few years it has been "later this year"
[1] https://amp.cnn.com/cnn/2021/10/28/cars/tesla-ntsb-cummings/...
Where'd he do this? Have a link? Curious.
Out of context, the tweet is largely innocuous as plenty of people might tweet such things. However, immediately following that tweet Missy Cummings was harassed ferociously by the Tesla shareholder base as seen here [1].
One of the leaders of the petition immediately following the tweet was Omar Qazi of WholeMarsBlog who is such a rabid fan and shareholder that he sees no issue with endangering children to support Tesla’s stock price as seen here [2] and the corresponding article. He also advocates for harassing any critics with vicious, baseless accusations [3].
Even if we assume that Elon is not directly organizing the attacks or by proxy, he has deliberately shaped a shareholding fanbase who will at the slightest mention. And, even if we assume that such a virulent base was purely accidental, which it never is, public figures have a moral duty to consider the likely consequences of their actions or speech.
His followers, many of which he engages with and responds to on Twitter such as the aforementioned Onar Qazi, are well known to viciously attack people he identifies as critics. These are not fringe elements, but the core of his fanbase. His actions are all but guaranteed to result in harassment of his critics, which is to his benefit, and he makes no effort to stop such interpretation, but rather harnesses it to suppress his opponents.
[1] https://twitter.com/MoodyHikmet/status/1451332679597625344
[2] https://twitter.com/WholeMarsBlog/status/1558122848601812993
[3] https://dawnproject.com/what-is-the-credibility-and-characte...
Cummings had a seat on the board of a LIDAR company and was paid by them [1].
She has had research funding from competitors in the industry Toyota, Uber, and Volkswagen [2].
She had associations with Tesla short-sellers, which raises questions about her impartiality [1].
Notably, she made public remarks suggesting an inclination to punch Musk [1].
Moreover, her legal team tried to suppress public discourse by attempting to get the petition on Change.org removed [1].
Also, should someone like this really hold an impartial position like the senior advisor of NHTSA? [3][4].
Let's not overlook these potential conflicts of interest just because they are inconvenient to acknowledge.
[1] https://wholemars.net/2021/11/21/missy-cummings-is-fighting-... [2] https://twitter.com/wholemarsblog/status/1450710771399200769 [3] https://twitter.com/heydave7/status/1450691154299731970 [4] https://twitter.com/TitterDaily/status/1450832510267629570
It sure is convenient how the blog post by Omar Qazi, a well known shareholder and propagandist who literally spends all of his time marketing for Tesla, made on November 21 fails to mention that event that occurred on October 25. Almost as if he has a massive conflict of interest and will say and do anything to support the value of his Tesla stock, such as endangering children [2]. I can not think of a more unreliable source other than Elon Musk himself (Elon Musk being a more unreliable source due to his even bigger conflict of interest).
And yes, a expert with decades of experience in safety engineering having a informed opinion supported by actual research they have done [3] on products is exactly someone who should have a position like senior advisor to NHTSA [4].
[1] https://www.nasdaq.com/articles/veoneer-says-cummings-to-res...
[2] https://twitter.com/WholeMarsBlog/status/1558122848601812993
[3] http://hal.pratt.duke.edu/sites/hal.pratt.duke.edu/files/u39...
[4] https://www.curbed.com/2021/10/elon-musk-missy-cummings-tran...
I am really skeptical at this point if it can ever master something to the point where we can safely trust our lives with it.
Another baffling feature was that the car refused to move after stopping at stop sign. Turns out there’s a “safety feature” to disable the engine if the seatbelt is not buckled. Thankfully I was just looking for parking and the only consequence was some inconvenienced cars behind me but that could get really bad in the wrong situation.
- if the person sitting in the middle has failed to buckle up or at least stretch the belt enough before the car started moving, now they can't do it until the car stops again;
- whenever the person in that place breathes out, they're running the risk of not being able to breathe back in next.
- when the car accelerates, there is a tendency for the person to weigh back into the seat. That causes the belt to contract.
Repeat the 2 latter points enough, and it becomes practically impossible to breathe or even to undo the buckle so you get trapped quite literally dying of asphyxia and in pain from the belt pressing hard against you.
* The fact that other cars of the same make and model do the same
* The fact that (at least) some other models of the same make do the same
* The fact that apparently I was the only person surprised at it when I mentioned to a few people who have cars of other makes as well
* The fact that it was like this when it was new
Perhaps none of these in isolation mean it's as designed but I reckon taken together they're a very good indication that it is as designed.
I just prefer to drive, I actually don't mind driving either.
However, there may be no other way to someday reach level 5 for general driving, other than to live with that middle zone for a while (keeping in mind that it’s already safer than humans driving alone) in order to conduct the training needed to reach level 5.
You don’t just wish, or regulate, level 5 into existence. You need to build toward it by having real systems deployed in the real world. Those real systems are not going to be level 5 at first or even level 3. This is where we are today, for general driving we only have level 2 and some level 3 if that.
Training is needed not only for the systems but also for drivers (including other drivers in traditional vehicles, they are also being trained unwittingly) and pedestrians, and regulators. Everyone is learning here. It’s not just an overnight, snap of the fingers kind of thing.
Gotta call bullshit on this. If human-piloted cars were bedecked in sensors, recording drivers' inputs and outputs, we could collect all the training we could need for L5 driving. It would take a ton of stringent curation, but that's going to be necessary anyway. It's not like Tesla has magically learned not to hit emergency vehicles stopped in traffic yet.
Funny thing that we will simultaneously make roads safer for all cars and pedestrians and cyclists too.
Or in other words, there are signals humans can send to other humans which can be understood even if you haven't seen that particular signal before, and those will be the biggest challenge for an AI driver, at least if it resembles the current generation of AI.
I did read the article. It’s ironic given her anti-Tesla stance how Tesla exactly follows what she recommends… “No hands-free L2” is pretty much exactly what Tesla urges and enforces its drivers to follow. Is the enforcement perfect? No, of course not; humans are simultaneously stupid and creative.
Wayne did a study several years ago IIRC and determined that even their trained, professional drivers being paid for this exact task could not reliably, effectively and quickly take over driving when a self-driving car made a mistake. It’s a task that humans are really quite bad at.
I know I am. Steering a car on an average road: not a big deal. Taking over on time when a new, borrowed Tesla tries to mess up on that same road: not so easy. Also, it takes more concentration than just steering the car normally.
"So, no L3 [full self-driving, but the driver must be ready to take the wheel], which is too confusing, and no hands-free L2 [partial self-driving]."
Let's zoom in on the part so you can read it, too:
"no hands-free"
You have your opinion about this, which you explain, but it differs from her opinion as expressed in the article.
Key point (read the quote):
She puts hands-on L2 inside the bounds of what she accepts.
That is also exactly what Tesla currently provides.
You also add a whole bunch of other confounding factors if you're not comparing same-driver accident rates. Accident rates vary by location, city/highway driving mix, driver attributes (age, gender, credit score, etc), and probably many other things that may be correlated with Tesla ownership.
I am a huge proponent for good self driving but experiencing driving assist and being a passenger in Tesla cars has made me much more cautious of the current capability.
Then it has to be exactly in the middle of the road, even in a two lane one with another car coming towards you.
Compared to what? I find this very hard to believe. Surely forward collision avoidance systems have stopped a number accidents over the years. To say that there is no evidence for it, no data even, seems to suggest that you might have a data problem. Hard to take them seriously when they don’t have good data, yet maintain their very negative position.
If an autonomous driving system clearly prevented or would have prevented a crash that happened to a human, then that's one life saved, is the philosophy. The minimum benefit from everyone using the system is at least one life.
But some people do not see it that way. So you are probably at cross purposes with them. I'm sure the skeptics understand there is evidence for what you think proves mitigation - just not for what they think would prove it.
I haven't got what it takes right now to describe at length the alternative way of looking at things, plus I could have it (or phrase it) wrong. Maybe someone else will comment.
Not something that can be told about the bulk of the drivers on the road.
Although I see the point of and agree with the necessity of 'crytal clear' situation, knowing what exactly can be expected from the assitance, I believe that the average driver should have as much help as possible. The roads are pretty unsafe yet, because of the human drivers.
I think roads and cars should be made to look less safe than they are, so the average driver will pay more attention.
Does it?
"Tesla's Autopilot AI team drives the future of autonomy of current and new generations of vehicles"
And if LIDAR is the wrong way to go for technical reasons, then if she pushes for better autonomous driving technology, she is working against her supposed interests.
She did divest preemptively, as described elsewhere in this thread.
We should believe Elon Musk who has the largest current conflict of interest in the world over a woman who has no current conflict of interest and deliberately took action to terminate any potential conflict of interest she may have had (to be fair that is legally required to hold such a post)?
Switching/using correct/monitoring autopilot modes is (somehow) still a major problem in aviation.
It’s not surprising that drivers (who basically get no training) have a hard time with this too, nor is it surprising that manufacturers are blinded by their own BS (see 737 Max for further reference).
Comically the GA airplanes with Garmin units in them were wildly better designed than the “professional FMS” boxes Honeywell sold. In particular, the flag ship “Honeywell Apex” units were terrible in comparison to the touch screen Garmins.
That’s wild, because the Honeywell box is fully integrated with the airplane and an order of magnitude more expensive.
There are something like 230 million licensed drivers.
So if I was the non-existent "average driver" with no control over risk, it seems to me I should expect a fatal accident roughly once every 100 lifetimes.
Or every 100 million miles.
I don't perceive the state of the art in autonomous driving as anywhere near that kind of reliability.
However… driving is one of the most dangerous things we do. 1/100 people (roughly) die in a traffic related accident in their lives. That’s bonkers and unacceptably high.
Well, that's your opinion.
My estimate of my risk of death is 100/100, so 1/100 does not seem that high to me.
What I think could legitimately be called "bonkers and unacceptably high" is the prevalence of medical errors as a cause of death in the US.
"Analyzing medical death rate data over an eight-year period, Johns Hopkins patient safety experts have calculated that more than 250,000 deaths per year are due to medical error in the U.S."
https://hub.jhu.edu/2016/05/03/medical-errors-third-leading-...
Since I don't drive drunk, and usually miss rush hour, I do think it's reasonable for me to consider going to the doctor the most dangerous thing I do.
I spent most of my adult life flying bush planes around Alaska and working to try to make things safer in that (horrifically unsafe) industry. Trust me when I say 1/100 is unacceptable and absurd in the extreme and humans are terrible at assessing risk objectively. Just your statement: “I don’t drive drunk and I usually miss rush hour” is indicative of misunderstanding how this works. That’s ok! Everyone does that, I did too until I was trained on specific ways to think about risk and risk mitigation strategies.
The risk often doesn’t come down to individual action - it’s links in the chain, or the Swiss cheese model of you prefer and when all the factors line up an accident is more likely.
It’s great that you don’t drink and drive, it is also great that you avoid driving at peak hours. But do you actively do risk mitigation before driving? Probably not, more power to you if you do, but probably not.
i guess this is a long way to say "something you are exposed to every day multiple times" is way different than a (albeit terrible) system you are exposed to when you are old and infirm and already prone to death.
still, everyone thinks the 1/100 won't be them so they feel it isn't a problem… that is a poor way to figure out "what should we do."
https://en.m.wikipedia.org/wiki/Existential_risk_from_artifi...
Many of the top AI experts are now coming out sounding warnings. AI Experts have predicted a chance of 10-20% human level extinction from AI, have a history of underestimating AI, but companies are rushing to the precipice, and it is just the early days.
Everyone should listen to this warning: https://m.youtube.com/watch?v=xoVJKj8lcNQ
But Elon's income on the other hand depends on the success of Tesla's FSD.
Well, his income and net worth have done pretty well without delivering FSD, so maybe not?
What would happen if he cancels FSD?