And the actual context is much less of a big deal than it seems: the biggest plausible consequence would be forcing Tesla to push an over-the-air update with better driver attention monitoring or alerting.
I encourage reading the actual report.
The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. NHTSA wisened up to that and independently purchased their own Tesla vehicles, and disabled software updates, so that they can reproduce those scenarios themselves.
If NHTSA asks Tesla to provide system validation tests showing that an updated version of their software meets the design intent of the system, Tesla would not be able to do so. If they can't prove the new Autopilot software corrects the safety-related defects identified in the current version, then it's not a valid recall remedy.
All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update.
The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing.
As of today, Autopilot IS dangerous software and it is not something that should be tested live on the streets.
So did the FAA and then they let Boeing self-validate the 737 MAX. Just saying..
If you read the report, you will realize that it says nothing about NHTSA might do if the kind of defect they are focussing on is confirmed.
It is certainly the kind of defect where it is plausible that better attention monitoring and alerting might be at least a partial mitigation, but that's about all you can reasonably conclude on that from from the report.
Why isn't Tesla prosecuted for that? It's lawless!
Software that controls multi-thousand pound machines at 70+mph isn't typical, and typical practices don't necessarily apply.
Cars will never be software, much like pacemakers and ICDs won't ever be software
Imagine an airbag incorrectly deployed sometimes, and the fix was to use a GPS geofence disable the airbag entirely on the test track, but only on days when the regulator was trying to reproduce spurious airbag deployments, not on crash test days.
Regulators were concerned after a car on non-FSD Autopilot (AKA Auto-Steer + Traffic Aware Cruise Control) hit an emergency vehicle parked half way in the right lane of a highway due to driver inattention. Tesla quickly pushed an update that uses ML to detect emergency lights and slow to a stop until the driver pushes on the accelerator to indicate it is clear to go.
That's not cheating, that's life-saving technology. No other steer assist technology gets (or sometimes is even capable of getting) updates that fast.
Contempt is such an overused tactic, and never meant anything anyway. Plus, it doesn't sound unrealistic to me.
What process is used to make such decisions?
Coaxing regulators into producing a test that can be optimized for is exactly how we got the WW scandal.
> What regulators need to do is keep up with the times.
Keeping up with the times sounds awfully like allowing insane things because some whiz kid believes there's no difference between a car and a website.
It's not typical software development in life-critical systems. If you think it is, you should not be working on life-critical systems.
So if a pharmacy swindles you out of your money or gives you fake drugs, I should reply 'that's just typical drug dealer'
If the NHTSA think there is a safety issue with Tesla Autopilot they will require Tesla to… fix it. Perhaps remotely.
Meh. I mean, I understand the emotional content of that argument, and the propriety angle is real enough. I really do get what you're saying. And if your prior is "Tesla is bad", that's going to be really convincing. But if it's not...
The bottom line is that they're getting close to 3M of these vehicles on the roads now. You can't spit without hitting one in the south bay. And the accident statistics just aren't there. They're not. There's a small handful of verifiable accidents, all on significantly older versions of the software. Bugs are real. They've happened before and they'll no doubt happen again, just like they do with every other product.
But the Simple Truth remains that these are very safe cars. They are. So... what exactly are people getting upset about? Because it doesn't seem to be what people claim they're getting upset about.
How regulators deal with this frankly tricky as the same will likely apply to all self driving systems.
“On average, there are over 5,891,000 vehicle crashes each year. Approximately 21% of these crashes - nearly 1,235,000 - are weather-related” “ 70% on wet pavement and 46% during rainfall. A much smaller percentage of weather-related crashes occur during winter conditions: 18% during snow or sleet, 13% occur on icy pavement and 16% of weather-related crashes take place on snowy or slushy pavement. Only 3% happen in the presence of fog.”
The statistics aren't there. The risks people have been shouting about for years just haven't materialized. If regulatory agencies are looking to reduce crashes, injuries, or deaths, there must be dozens of more effective places to focus attention on than Autopilot. But it's 2022, and yet again, it's on the front page of Hacker News, and the top comment is (you guessed it): the naming of the features is the problem.
It's Groundhog Day all over again. Geesh.
Unless this is on the same road and conditions, instead of “autopilot where and when used vs. real drivers everywhere and everywhen” it is meaningless, even moreso if it doesn't also account for “autopilot disengages immediately before anticipated collision so it doesn't count as driving when it occurred.”
People count disengagements directly before collisions such as NTSB is doing in the article. Where people disagree on how wide that window should be. Disengaging 15 seconds before a collision is hardly autopilots fault, but even picking such a wide threshold doesn’t somehow push autopilot to less safe than the average driver.
This is just so frustrating. It's not meaningless, it's measured data. Could it be better corrected? Could there be other analysis done? Sure. But data is data, and the effect is extremely large. Cars with AP enabled aren't just safer, they're like 5x safer!
You can't wave that away with a innumerate statement about confounding factors. You need to counter data with data, and (despite millions of Teslas on the road now!) no one has it.
Is it really so hard to just accept that... the system is safe?
Measured data that is used to make a comparison to data not gathered under similar conditions aside from the difference being assessed or structured so as to support controlling for the irrelevant differences is, in fact, meaningless for that purpose.
It may have meaning in other contexts, but when it's offered to justify the comparison it cannot support, it is, in that context, meaningless.
Is autopilot engaged in the places where crashes are frequent, eg. during left turns?
What are the “scenario-equalized” safety stats for autopilot vs human drivers?
It seems reasonable for a regulator to decide what that time span is and require all automated driving assist systems to report in a consistent way. I'm curious what % of the time a crash in a typical car occurs within N seconds of cruise control or lane assist or traffic aware cruise control engaged.
Further, rather than what the article is insinuating autopilot disengages when users apply the break such as occurs when they are trying to avoid an accident. What’s concerning is cases when autopilot decides to give up control and the driver isn’t ready to take over.
No way would I sign, and they'd fix it, or see me in court.
And not rich guy wins US court, but Canadian court. And yeah, it's different.
So people invested in the argument now have to resort, like you just did, to theorizing about a literal conspiracy to hide the data that you know must be there even though it can't be measured.
It's just exhausting. They're fantastic cars. Get a friend to give you a ride.
Tesla publishes this data quarterly. And it's been largely unchanged for years. And yet we still keep having these discussions as if this system "can't be proven safe" or "is presumptively unsafe" despite years of data showing the opposite.
It's just getting so tiresome. Where are the accidents if it's unsafe? How are they managing to hide all the bodies?
[1] Now, could there be a more rigorous study? Undeniably! But no one has managed to do one, despite this kind of data being readily available (especially to bodies like the NHTSA).
The entire rest of the industry has 1 fatality. Tesla has dozens, and 14 of those are old enough (and located in the right country) to be part of this investigation. (The multiple Tesla autopilot/FSD fatalities from 2022, including the 3 from last month, are not part of this investigation.)
What is this evidence?
I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models.
Not to disparage Andrej, sometimes (frequently, even) what executive leadership thinks is going is not the day-to-day reality of the team.
It’s all smoke and mirrors. You cannot perform proper validation of AI systems. Rollbacks of new versions of ML models are very common in production, and even after very extensive validation you can see that real life results are nothing like what tests have shown.
You basically put another ML on top of ML, to correct it. I’ve seen that in use in production systems, and it helps with some problems and generates new ones. And if you thought that reasoning about correctness was hard before…
And what do you mean by disabling AI, if input wasn’t in the training set? That’s the whole point of ML, to reason about new data based on data seen in past.
I think we like to think this is true.
In reality, I have seen a lot of real world ML models. I wouldn't trust ANY of them to do extrapolation. There are just tons of real world problems, and extrapolation is HARD.
I have to put extremely tight boundaries on ML models for deployment scenarios, and ALWAYS have a backup rule engine in case the ML model comes up with an answer that has a low confidence score.
> How do you identify the outlier? You need to write some rules that could look at it. But that’s a lot of rules. What if you could use computers to do that?
> You need to write some rules that could look at it.
Pretty much. Any time ML is involved, you will need TONS of lines of code.
In short, tightly define the target classes your ML model deals with.
Any variable that falls outside your tightly bound list of target classes, you have to deal with using a rules engine. THEN you need to spend a lot of time doing work to minimize false positive classification in your target classes.
And make sure that "false positive, high confidence" classifications don't do racist things/lose the business a lot of money things.
ML projects are just a ton of work. You essentially make the entire ML workflow, and you NEED a backup "not-ML" workflow.
In my experience, 50-80% of normal software engineering projects fail.
90% of ML projects fail. Square the fraction of normal software projects.
ML is complex AND it's a ton of work. Really, really hard.
Give it a few years and they will probably introduce LIDAR.
>> What is this evidence?
Without a documented development and testing program, every development is essentially this.
I think the onus should be on Tesla to prove that their testing and validation methodology is sufficient. Until and unless they have done so, Autopilot should be completely disabled.
I really don't get why the regulatory environment is so behind here. None of these driver assistance technologies (from any manufacturer, not just Tesla) should be by default legal to put in a car.
Wow. It seems like now on HN, this sort of turn-of-phrase has sadly become a boiler plate way to dismiss a host of comments. IE, what "actual report" are you referring to?? The headline article is about the investigation that has opened but not yet closed, as the linked text shows (it's a PDF but it's essentially a press release showing discoveries so far - why they've escalated. It's not closed and not friendly to Tesla).
"Accordingly, PE21-020 is upgraded to an Engineering Analysis to extend the existing crash analysis, evaluate additional data sets, perform vehicle evaluations, and to explore the degree to which Autopilot and associated Tesla systems may exacerbate human factors or behavioral safety risks by undermining the effectiveness of the driver’s supervision"
I see no basis for this conclusion, which appears to be pure speculation about what NHTSA might decide is necessary and sufficient to address the potential problems if confirmed by the deeper analysis. I encourage reading the actual report.
> I encourage reading the actual report.
I did, and the conclusion do which you appeal to it does not appear to be well-supported by it.
You cannot OTA HW deficiencies.
Now, will they be forced to have monitoring like that, to be on par with their competitors? That’s a different story, and given how weak USA regulatory agencies are, and how reckless Tesla is at disregarding them - I’m pretty sure Tesla won’t be hurt by it.
I saw a similar reaction to Volvo's announced plans for such tech (it was a YouTube video) - it seemed to be a completely offline system, but people still read "camera" and reacted that they don't want "someone watching them" in the car.
But there's no such system, nor is there any trivial way of demonstrating that such a system works that way. Because all car companies and insurance companies want data.
But for systems closer to AP (that claims to stop at the lights, take highway exits, etc), it’s de facto industry standard to have dedicated camera for monitoring.
See. That has been my whole point for months that both Autopilot and FSD is still unproven safety critical software and it goes to show that if used in circumstances say at night it becomes even far dangerous to use at the worst time to drive, like I have said before [0][1][2][3]. Even worse that it lacks proper driver monitoring.
I guess they should be required to have this driver monitoring hardware installed on their cars as well as night vision cameras to avoid the crashes I have listed below. If the regulators enforce this, perhaps they might have saved another Tesla driver from losing control or avoided another crash.
Or perhaps if Tesla still finds it difficult to prevent further crashes using night vision cameras, perhaps they have to admit that they should have used LIDAR instead.
[0] https://news.ycombinator.com/item?id=29639080
[1] https://news.ycombinator.com/item?id=30267710
Its pretty much not worth even paying attention to.