Watch This Giant Teddy Bear “Drive” a Tesla (Into Child Mannequins)
latimes.com
latimes.com
Similar test in 4K: https://vimeo.com/851340215/cb9acd120b?share=copy
Advanced driver assistance systems (ADAS) are only safe while operated by an attentive driver. Unfortunately, research shows that ADAS can not be used without inducing driver complacency.
To combat this, driver monitoring systems (DMS) are used to detect inattention and loudly return control back to the user to prevent unsafe operation.
The tests show that in the presence of constant wheel torque used to simulate hands on the wheel (via a weight attached to the wheel), Tesla’s facial DMS is unable to determine that a giant teddy bear is not an attentive driver.
In the other video linked, it similarly shows that it can not detect an individual texting while wearing sunglasses amongst other similarly common scenarios.
The inability to handle the case of “hands on wheel, teddy bear” which is a softball test compared to “hands on wheel, inattentive driver” proves a gross systemic deficiency in Tesla’s approach to safety-critical system design.
The inability to handle or even consider trivial cases, common cases such as sunglasses, and their policy of default-allow for unsafe practices are a staggering example of safety engineering incompetence and malfeasance.
You clearly believe you know of more effective safety engineering methods.
Please enlighten us.
I've only used cruise control on my non-electric car. The first thing I noticed is that I no longer needed to check my speed, leaving me more free time to look out the window.
As I understand it the ADAS systems not only provide cruise control, they provide lane-keeping and distance-keeping. That will likely reduce the number of drunk driving, text driving, and sleepy driver accidents.
Any system can be fooled. I could continue to use my cruise control, set at 70, to take a 15mph exit ramp.
This is a safety-critical system. Avoidable deaths occur when it is designed inappropriately. There have already been 38 deaths using the Tesla ADAS; the majority of which occurred because drivers are lulled into complacency and become chronically inattentive doing actions such as watching a video.
A proper DMS could have avoided these deaths caused by ADAS induced complacency. It is completely unacceptable that 5 years after killing a driver watching a video in the drivers seat that it still fails to detect such misuse that is documented to occur and which is easily solved.
A proper DMS should only allow ADAS operation with a positively identified attentive driver. It must fail-safe.
Such a system would trivially reject a teddy bear in the seat regardless of constant torque on the wheel meant to minimally simulate a driver holding the wheel. An inattentive driver holding the wheel is a strictly harder problem in both facial and wheel detection. This is not a “trick” or a “bypass”, this is a T-ball and Tesla struck out.
Any properly designed solution must demonstrate the ability to solve all trivial problems to support the conclusion that all harder problems have been solved robustly. You would not trust a person to do calculus if they can not add. You should not trust a DMS if it can not reject a absent driver even if it sometimes works in harder cases. The lack of awareness needed to fail or not even attempt to solve trivial cases demonstrates systemic incompetence.
To judge incompetence you need to be able to show what competent behavior is. Please be specific and clearly show what will, and HOW to, achieve competent behavior in an ADAS system. Tesla's approach to that problem is that the human is always responsible. It says so "right on the can". Write your "competent" code as open source and post it for all to use.
As for malfeasance, you need to show mens rea, the intention of wrongdoing, aka a "guilty mind". Please provide specific information you have that anyone at Tesla intended to cause harm. Name names. Be aware that falsely claiming malfeasance by an individual might be legally actionable defamation.
Safety critical systems fail. Nuke plants melt down, commercial and military aircraft crash, ...
ADAS saves lives, more lives statistically than average but not all lives.
You want perfection. You can't have it.
It seems wrong to cast aspersions on people at Tesla without showing how "you would do it right". Be sure to include a github link in your next reply so I can download and test your coded solution.
1. Competence is judged in accordance with an acceptance criteria. You do not need to be able to show competent behavior to deem something incompetent. You are not competent to develop a fusion reactor. Nobody can do that, so we know you can not and are thus incompetent. You are not competent to develop a manned lunar space mission. It was done historically, but not now, so we know you can not and are thus incompetent. You are not competent to develop a rocket that can reach GEO. Others can do it, but we know you can not and are thus incompetent.
2. A competent DMS must only allow ADAS operation when it can positively identify a human driver looking at the road in a manner comparable to a attentive driver. Before you go on about some sort of inane tangent demanding I, a non-expert on gaze monitoring, define the exact parameters of what constitutes a "attentive driver" and then seek loopholes in whatever criteria I suggest to declare "victory" over safety, the acceptance criteria I just put forward explicitly does not absolve the manufacturer of any liability for deficiencies in their design whether known or unknown. The use of unproven technology in safety-critical contexts does not get a free pass. It must be deployed with the utmost care until an adequate analysis of its safety properties can be determined.
3. Although I can not speak to what, specifically, constitutes a successful implementation, I can speak to what is an immediate failure. In much the same way that I can not make a rocket that reaches LEO, but can determine that a rocket that explodes on the launchpad is not successful, I can determine what is a clear failure. Allowing the ADAS to operate while there is no human in the driver seat is an immediate, egregious failure. It is completely unacceptable on its face even if there were no known solution. Even worse, human face detection is an undergraduate computer vision project. Basically every phone and camera in the world has human face detection. If a human face is not detected for a certain amount of time then the ADAS should relinquish control. This may cause excessive disengagement if your human face detector is poor, but the reverse fails unsafe which is unacceptable. Literally anybody with even the minimum of safety design training would know that.
4. You demand that I solve Tesla's problem, that their hundreds of paid engineers can not solve, for free and then post it as open source. For some reason you do not demand Tesla, the manufacturer of the product and a nearly trillion dollar company, to release their source code for evaluation before it is allowed to endanger the lives of the public. For that matter, Tesla does not disclose their code even to the regulatory agencies, yet you demand a random member of the public to engage in a more thorough and transparent process than the company with billions of dollars who is attempting to deploy an incomplete safety critical product on the public roadways. The double standard is appalling.
5. It is not necessary to specifically identify individuals to declare malfeasance. I do not know who was in charge of Philip Morris when they deliberately obfuscated evidence linking smoking to lung cancer, but I know that they engaged in malfeasance.
6. The actions of Tesla demonstrate such a disregard for accepted practices in the development of safety-critical products and are such a gross deviation from the normal standard of care and occur with such regularity that it is improbable that the actions are mere criminal recklessness. Either they have no staff who are trained in safety design or they override them to ship product. Both are unacceptable.
They are developing a safety-critical product. Their management knows they require qualified safety experts. If they choose to retain no such individuals, then they have intentionally engaged in conduct that will lead to no safety oversight despite knowing that they should. If they have safety experts and override them, it is self-explanatory how that is unacceptable.
7. You demand I specifically name individuals while also declaring that your demand incurs legal consequences on me while accepting no increased burden yourself. Your attempts to raise the standard of proof exclusively on your opponents and then hurl baseless legal threats are anti-social debate tactics. How about you make a notarized statement under penalty of perjury that you know Tesla is following adequate safety procedures, and if they are not then you are intentionally lying to harm the public. If Tesla is following adequate safety procedures, then the second clause has no effect. If Tesla is not following adequate safety procedures, then you are trivially guilty of perjury.
Elon Musk is personally guilty of manslaughter. For the same reasons as above. In addition, Elon Musk was publicly informed that Tesla ADAS would not stop for a school bus with the stop arm extended, as is required by law, in 2022-10 [2]. He was separately informed again in 2023-02 [3]. He took no action to correct the behavior or to recall the Tesla ADAS until the behavior is corrected. On 2023-03-15, a Tesla with ADAS engaged hit a minor exiting a school bus with the stop arm extended [4] causing serious injuries. This is recorded in the NHTSA SGO database [5] as Report ID 13781-5100 which definitively confirms that the ADAS system was operating at the time of collision. Elon Musk was informed that the Tesla ADAS would engage in grossly illegal and reckless behavior and then it did that behavior and caused serious personal harm to a member of the public. He has been separately informed of numerous other life-threatening safety defects and exactly zero corrective action has been taken to mitigate them.
9. Safety critical systems do fail. This is despite great efforts and care taken to rigorously tackle safety problems, the decades of experience in developing high quality safety protocols, and a solemn regard for the challenging task of keeping humans safe. Tesla throws all of this away and tackles problems with a callous disregard for human life and a "ends justify the means" philosophy that is incompatible with safety. Instead of transparency, they deliberately obfuscate their raw data, allow no independent evaluation or auditing, and then publish bogus safety reports like this [6] which include no actionable data at all that are meant to mislead customers into trusting unsafe systems with their lives. They do not even include the count of crashes used or the number of miles driven. They publish a single ratio and call that a safety report. In contrast, Waymo published this safety report [7] at 1,000,000 driverless miles. It exhaustively lists, describes, and analyzes every crash that occurred. It discusses methodology, comparisons, confounding factors, and all the other basic things expected from a real safety report.
Waymo is registered with the CA DMV to do driverless testing [8] and reports all vehicles, mileage, and contact events to the DMV. Tesla is not registered to do driverless testing and has not done any safety driver testing despite claiming their customers are "safety drivers" doing testing. They deliberately misclassify their Full Self Driving, L4, Beta program as a L2 Beta program to avoid mandatory reporting requirements that would reflect badly on them.
10. You can present exactly zero audited evidence that Tesla ADAS saves lives. The only support for that position is the unaudited Tesla marketing statements I pointed to above drawn from private data that Tesla does not allow anyone access to. Unaudited reports by the manufacturer are literally worthless. They have the largest possible conflict of interest to lie or misrepresent the data. All publicly available evidence points to the opposite with Tesla reporting to NHTSA over 700 accidents with no known survivors attributable to FSD in the SGO database over an estimated 350M miles. That is over 150x more deadly than the average human driver.
11. I do not want perfection. I want proven, audited safety statistics, robust safety processes that minimize avoidable deaths and risks and ratchet toward improved safety, and exhaustive safety testing to avoid safety regressions. Tesla provides exactly none of these. I would also like the people making safety-critical systems to have a respect for the humans lives they are putting at risk. Tesla and Elon Musk do not have that either.
In conclusion, your opinions are wrong in every conceivable way. Literally every single statement derived from your beliefs is harmful and detrimental to safety design and has been repudiated by decades of work by actual safety experts making products fit to depend on. And your argumentation style is anti-consumer and anti-social as you seek to burden others with proving your unsupported positions wrong to a ridiculous standard of proof that you yourself are unwilling to attempt and then threaten them in an attempt to cow them.
[1] https://www.documentcloud.org/documents/23574198-elluswamy-d...
[2] https://dawnproject.com/tests-reveal-that-tesla-fsd-will-dri...
[3] https://www.youtube.com/watch?v=_ZiSZbWIrzA
[4] https://www.rflaw.net/howard-gene-yee-hits-10th-grader-halif...
[5] https://static.nhtsa.gov/odi/ffdd/sgo-2021-01/SGO-2021-01_In...
[6] https://www.tesla.com/VehicleSafetyReport
[7] https://storage.googleapis.com/waymo-uploads/files/documents...
[8] https://www.dmv.ca.gov/portal/vehicle-industry-services/auto...
Are you confusing ADAS with Full Self Driving? They are nowhere the same.
An ADAS system, like my cruise control, won't stop for a school bus, for example.
Do you insist that my cruise control shut off if I look away from the road? It currently doesn't.
I rented a Tesla 3 for a week. The ADAS system worked as expected. One instance does not a pattern make but I was quite impressed. I have a lifelong involvement with vision and robotics and understand how difficult it is to "follow the lane" in all kinds of lighting, weather, and traffic. In some driving conditions even people can't follow the lane.
As for engineering safety systems we had many debates about protecting human/robot interactions. This is the reason you see fences around robots in factories. Opening the gate will shut down the robot and that will shut down the production line. Workers have still been killed by being inside the fence despite all safety interlocks and worker training.
You claim to want "a ratchet toward improved safety". It seems to me that these early attempts at driver awareness IS a ratchet toward improved safety as they did not exist before. Ratchet safety requires failures and corrections. You claim that "Tesla provides none of these" but seem to ignore the new, previously non-existing, attempts at driver awareness.
Teslas are reasonably safe cars. To quote Euro NCAP safety rating results:
Following this test, Model S again received a 5-star safety rating and the highest Overall Score among any vehicle tested in this protocol. It also performed exceptionally well in more detailed tests across the board: 98% in Safety Assist. 91% in Child Occupant Protection.
Sandy Monro, an auto expert, did a review of the safety design of Tesla. Monro pointed out the engineering features that preserved passenger compartment integrity in a collision. The Tesla I rented had steering wheel airbags, side-curtain airbags, and driver/passenger separation airbags. My current car only has steering wheel / passenger airbags.
Even if you insist on confusing ADAS with FSD the driver is "pilot in command" and fully responsible for everything all the time. If there is an accident it is the drivers fault. Until there are no drivers that will always be true. I am unaware of Tesla claiming that you don't need a driver.
Having worked on an ambulance corp for 5 years I can state from personal experience that people die in car accidents. That will continue as long as there are cars.
You seem so passionate about this topic with a particular focus only on Tesla. That strikes me as odd. There are many car companies with ADAS systems and ADAS failures, most of which don't include an attempt at driver awareness. Where is your review of other systems? Comma, for example, at the Comma Con 2 conference explicitly said they are NOT liable for anything.
If ADAS/FSD safety is your issue please post safety issues of non-Tesla cars. There are 91 car companies in China with ADAS and most of the worldwide majors have ADAS. So lets have a more general, non-Tesla safety discussion.