Hacker News Reader

Comment by dangrossman | Hacker News Reader

The worst case outcome would require Tesla to disable the autopilot suite entirely, for an indeterminate amount of time, perhaps permanently on the existing fleet of vehicles.

The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. NHTSA wisened up to that and independently purchased their own Tesla vehicles, and disabled software updates, so that they can reproduce those scenarios themselves.

If NHTSA asks Tesla to provide system validation tests showing that an updated version of their software meets the design intent of the system, Tesla would not be able to do so. If they can't prove the new Autopilot software corrects the safety-related defects identified in the current version, then it's not a valid recall remedy.

All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update.

View on HN

> All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update.

What is this evidence?

I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models.

[0] https://youtu.be/hx7BXih7zx8

I’ve been working few years ago at a very big tech company, focusing on validation of the AI systems.

It’s all smoke and mirrors. You cannot perform proper validation of AI systems. Rollbacks of new versions of ML models are very common in production, and even after very extensive validation you can see that real life results are nothing like what tests have shown.

Can't you do outlier detection, and disable the AI if the input wasn't in the training set?

How do you identify the outlier? You need to write some rules that could look at it. But that’s a lot of rules. What if you could use computers to do that?

You basically put another ML on top of ML, to correct it. I’ve seen that in use in production systems, and it helps with some problems and generates new ones. And if you thought that reasoning about correctness was hard before…

And what do you mean by disabling AI, if input wasn’t in the training set? That’s the whole point of ML, to reason about new data based on data seen in past.

> That’s the whole point of ML, to reason about new data based on data seen in past.

I think we like to think this is true.

In reality, I have seen a lot of real world ML models. I wouldn't trust ANY of them to do extrapolation. There are just tons of real world problems, and extrapolation is HARD.

I have to put extremely tight boundaries on ML models for deployment scenarios, and ALWAYS have a backup rule engine in case the ML model comes up with an answer that has a low confidence score.

> How do you identify the outlier? You need to write some rules that could look at it. But that’s a lot of rules. What if you could use computers to do that?

> You need to write some rules that could look at it.

Pretty much. Any time ML is involved, you will need TONS of lines of code.

In short, tightly define the target classes your ML model deals with.

Any variable that falls outside your tightly bound list of target classes, you have to deal with using a rules engine. THEN you need to spend a lot of time doing work to minimize false positive classification in your target classes.

And make sure that "false positive, high confidence" classifications don't do racist things/lose the business a lot of money things.

ML projects are just a ton of work. You essentially make the entire ML workflow, and you NEED a backup "not-ML" workflow.

In my experience, 50-80% of normal software engineering projects fail.

90% of ML projects fail. Square the fraction of normal software projects.

ML is complex AND it's a ton of work. Really, really hard.

> What is this evidence?

I think the onus should be on Tesla to prove that their testing and validation methodology is sufficient. Until and unless they have done so, Autopilot should be completely disabled.

I really don't get why the regulatory environment is so behind here. None of these driver assistance technologies (from any manufacturer, not just Tesla) should be by default legal to put in a car.

>> They're flying by the seat of their pants, introducing potentially lethal regressions in every update.

>> What is this evidence?

Without a documented development and testing program, every development is essentially this.

I see your point, to OP's point, I know a couple people who were horrified at what they saw and it did not match this public talk. Both started at least 6 months after this video, and both left Tesla within 8 months, of their own volition. Unfortunately, off the record.

Not to disparage Andrej, sometimes (frequently, even) what executive leadership thinks is going is not the day-to-day reality of the team.

can confirm, a former coworker had just come from Tesla 5 years ago and he had serious ethical problems with his work over there. Tesla is killing people through negligence and greed, it's pretty disgusting, but par for the course

This is the Karpathy that gave a big talk about how vision was superior to radar when Tesla dropped all radar units at the height of the chip crisis. Now they are bringing radar back.

Give it a few years and they will probably introduce LIDAR.

Tesla is bringing Radar back? First I've heard about it, and good news if true.

https://electrek.co/2022/06/08/tesla-files-use-new-radar-con...

Wasn’t this approval based on an application from 2018?

> The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated.

Why isn't Tesla prosecuted for that? It's lawless!

No, that’s typical software development. Find a bug in circulation, fix and deploy a fix. There are probably hundreds of internal issues that get fixed per normal protocol, as with any piece of software. Putting out a “recall” or alert for every modification to the code is pointless. What regulators need to do is keep up with the times. They need to have their own safety test suite which manufacturers can test against, and be independently audited

> No, that’s typical software development.

Software that controls multi-thousand pound machines at 70+mph isn't typical, and typical practices don't necessarily apply.

Yes, those practices absolutely shouldn't apply for self driving cars. Good luck regression testing how a system change impacts the AI handling of every edge case of traffic.

Waymo does this. Their infrastructure costs would be astronomical though if not attached to a company with its own cloud

It seems like the test suites for these deep learning based models are themselves almost comprehensive knowledge bases from which you could build a more traditional control software.

> They need to have their own safety test suite which manufacturers can test against

Coaxing regulators into producing a test that can be optimized for is exactly how we got the WW scandal.

> What regulators need to do is keep up with the times.

Keeping up with the times sounds awfully like allowing insane things because some whiz kid believes there's no difference between a car and a website.

It's typical software development when there's nothing at stake (such as human life). When human life is at stake, the controls on software reviews/changes/updates SHOULD be much tighter, but as there's no governing body to require this, it's on the developers themselves to do it right. Tesla is an example of a company that does not employ those controls in mission/life critical software.

Sorry, typical software development is for a national regulator to find some illegal feature in your website, and then you disable the feature for specific IP ranges or geofenced areas where the regulator's office is? No, I don't think it is.

Typical software development as practiced by Uber, perhaps

Yeah, well, just like half of US sites just flatly block Hetzner IPs (where I happen to have a VPN) because GDPR.

They've been accused of silently pushing updates to fix specific scenarios in order to gaslight the regulators.

Imagine an airbag incorrectly deployed sometimes, and the fix was to use a GPS geofence disable the airbag entirely on the test track, but only on days when the regulator was trying to reproduce spurious airbag deployments, not on crash test days.

That sounds like a cartoon villain. Here's an actual example:

Regulators were concerned after a car on non-FSD Autopilot (AKA Auto-Steer + Traffic Aware Cruise Control) hit an emergency vehicle parked half way in the right lane of a highway due to driver inattention. Tesla quickly pushed an update that uses ML to detect emergency lights and slow to a stop until the driver pushes on the accelerator to indicate it is clear to go.

That's not cheating, that's life-saving technology. No other steer assist technology gets (or sometimes is even capable of getting) updates that fast.

> That sounds like a cartoon villain.

Contempt is such an overused tactic, and never meant anything anyway. Plus, it doesn't sound unrealistic to me.

What's to stop them to push an update that turns the workaround off because it leads to unexpected deceleration in odd lighting, or when there is an emergency vehicle on the other side of a divided freeway?

What process is used to make such decisions?

Evidence they are doing anything like this? Or are they fixing the "actual" issue.

Per the article, the regulators are sufficiently concerned that they're blocking all updates to their test vehicles.

No, it is not. You sure don't do it in aerospace. Each change is verified, the entire system is validated prior to a release.

> No, that’s typical software development.

It's not typical software development in life-critical systems. If you think it is, you should not be working on life-critical systems.

Releasing software updates in normal is life-critical systems. Can't believe you are arguing differently.

Narrator: (s)he doesn’t

> typical software development.

So if a pharmacy swindles you out of your money or gives you fake drugs, I should reply 'that's just typical drug dealer'

It’s not typical for safety critical systems. A car isn’t a Web 3.0 app and shouldn’t be updated in the same way.

> No, that’s typical software development.

Cars will never be software, much like pacemakers and ICDs won't ever be software

It really is insane. It’s one thing to have flaws, it’s quite another to stealth cover-them-up like it’s a game of hide and seek.

Wut? You want Tesla prosecuted because they are fixing issues over the air?

If the NHTSA think there is a safety issue with Tesla Autopilot they will require Tesla to… fix it. Perhaps remotely.

If you read the report, you will realize that NHTSA is considering requiring Tesla to do better driver attention monitoring, or to improve alerting. They are not considering banning autopilot.

I assure you, I'd already read the report before it was shared here. I also assure you, there's more to the investigation than that.

Perhaps, but that’s speculation at best.

But you’ve been assured with absolutely no evidence /s.

99% of everything is speculation at best, even this site is pretty much (high value) speculation as a service.

The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing.

As of today, Autopilot IS dangerous software and it is not something that should be tested live on the streets.

> The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing.

So did the FAA and then they let Boeing self-validate the 737 MAX. Just saying..

Yup, that's true. Let's see.

> If you read the report, you will realize that NHTSA is considering requiring Tesla to do better driver attention monitoring, or to improve alerting

If you read the report, you will realize that it says nothing about NHTSA might do if the kind of defect they are focussing on is confirmed.

It is certainly the kind of defect where it is plausible that better attention monitoring and alerting might be at least a partial mitigation, but that's about all you can reasonably conclude on that from from the report.

> introducing potentially lethal regressions in every update.

Meh. I mean, I understand the emotional content of that argument, and the propriety angle is real enough. I really do get what you're saying. And if your prior is "Tesla is bad", that's going to be really convincing. But if it's not...

The bottom line is that they're getting close to 3M of these vehicles on the roads now. You can't spit without hitting one in the south bay. And the accident statistics just aren't there. They're not. There's a small handful of verifiable accidents, all on significantly older versions of the software. Bugs are real. They've happened before and they'll no doubt happen again, just like they do with every other product.

But the Simple Truth remains that these are very safe cars. They are. So... what exactly are people getting upset about? Because it doesn't seem to be what people claim they're getting upset about.

Total straw man. The question isn't whether Tesla's are safe to drive. The question is whether "autopilot" is safe to auto pilot.

I think that's largely a correct way to reason about it. And what data we have says that a Tesla on autopilot is safer than a Tesla driven by a human, which is safer than an average vehicle. Both by quite some margin.[1]

Tesla publishes this data quarterly. And it's been largely unchanged for years. And yet we still keep having these discussions as if this system "can't be proven safe" or "is presumptively unsafe" despite years of data showing the opposite.

It's just getting so tiresome. Where are the accidents if it's unsafe? How are they managing to hide all the bodies?

[1] Now, could there be a more rigorous study? Undeniably! But no one has managed to do one, despite this kind of data being readily available (especially to bodies like the NHTSA).

Actual real world data says autopilot is on average at least as safe as the average driver in the average car. Of course that’s on average, in many specific situations it’s much worse but conversely it means it’s better in other situations.

How regulators deal with this frankly tricky as the same will likely apply to all self driving systems.

Those real world autopilot averages happen exclusively in the most trivial driving situations. Fair weather and almost exclusively on limited access roads. No real world average driver dataset exists that is similarly restricted to the subset of least error-prone driving situations.

But it is also not the case that people are crashing and dying left and right from the FSD beta, or as a result of using Autopilot in less than ideal conditions. This despite OTA updates increasing functionality and millions more cars being sold. Even if what you're saying is true: it is empirically true that in practice, humans take over when necessary.

The statistics aren't there. The risks people have been shouting about for years just haven't materialized. If regulatory agencies are looking to reduce crashes, injuries, or deaths, there must be dozens of more effective places to focus attention on than Autopilot. But it's 2022, and yet again, it's on the front page of Hacker News, and the top comment is (you guessed it): the naming of the features is the problem.

It's Groundhog Day all over again. Geesh.

Fair weather limited access highways is one of many datasets available. However, Autopilot operates in wet and rainy conditions so that’s hardly an accurate assessment. Weather bad enough to prevent autopilot is a contributing factor ~5% of accidents.

“On average, there are over 5,891,000 vehicle crashes each year. Approximately 21% of these crashes - nearly 1,235,000 - are weather-related” “ 70% on wet pavement and 46% during rainfall. A much smaller percentage of weather-related crashes occur during winter conditions: 18% during snow or sleet, 13% occur on icy pavement and 16% of weather-related crashes take place on snowy or slushy pavement. Only 3% happen in the presence of fog.”

https://ops.fhwa.dot.gov/weather/q1_roadimpact.htm

> Actual real world data says autopilot is on average at least as safe as the average driver in the average car.

Unless this is on the same road and conditions, instead of “autopilot where and when used vs. real drivers everywhere and everywhen” it is meaningless, even moreso if it doesn't also account for “autopilot disengages immediately before anticipated collision so it doesn't count as driving when it occurred.”

The point is people have risk tolerances, if the average driver is taking an acceptable risk then suggesting a lower risk than that is unacceptable is hardly reasonable. If that level of risk is actually unacceptable then you should be suspending peoples licenses for doing 5 MPH over the speed limit etc. Instead driving laws are based on a wider risk tolerance.

People count disengagements directly before collisions such as NTSB is doing in the article. Where people disagree on how wide that window should be. Disengaging 15 seconds before a collision is hardly autopilots fault, but even picking such a wide threshold doesn’t somehow push autopilot to less safe than the average driver.

> Unless [...] it is meaningless

This is just so frustrating. It's not meaningless, it's measured data. Could it be better corrected? Could there be other analysis done? Sure. But data is data, and the effect is extremely large. Cars with AP enabled aren't just safer, they're like 5x safer!

You can't wave that away with a innumerate statement about confounding factors. You need to counter data with data, and (despite millions of Teslas on the road now!) no one has it.

Is it really so hard to just accept that... the system is safe?

> This is just so frustrating. It's not meaningless, it's measured data.

Measured data that is used to make a comparison to data not gathered under similar conditions aside from the difference being assessed or structured so as to support controlling for the irrelevant differences is, in fact, meaningless for that purpose.

It may have meaning in other contexts, but when it's offered to justify the comparison it cannot support, it is, in that context, meaningless.

Only if the differences between conditions are enough to matter.

If autopilot disengages 1 second before crashing into a stationary object, does this count as autopilot crashing or the human driver?

Is autopilot engaged in the places where crashes are frequent, eg. during left turns?

What are the “scenario-equalized” safety stats for autopilot vs human drivers?

Tesla's statistics count it as autopilot if it was engaged within 5 seconds of the collision.

It seems reasonable for a regulator to decide what that time span is and require all automated driving assist systems to report in a consistent way. I'm curious what % of the time a crash in a typical car occurs within N seconds of cruise control or lane assist or traffic aware cruise control engaged.

The article says they're using a 1 second threshold, not 5, and that a substantial number of accidents fall between the two numbers.

No, those 16 accidents are counted as a failure by autopilot. Hell the NTSB is explicitly doing so in the article.

Further, rather than what the article is insinuating autopilot disengages when users apply the break such as occurs when they are trying to avoid an accident. What’s concerning is cases when autopilot decides to give up control and the driver isn’t ready to take over.

By actual real world data, you mean cherry picked average data published by Tesla, that doesn’t account for any bias, and wasn’t audited by independent third parties?

Sources for claims would be appreciated.

It’s also worth noting Tesla is nearly infamous at this point for making owners sign NDAs in exchange for repairs when their autopilot is likely at fault.

This is a meme. I've never seen any significant corroboration on this. I mean, how would they even know without violating their own published privacy policy? I think you got hoodwinked. This is what's so frustrating about this argument. The clear truth is that the system is operating very safely on millions of vehicles, because a safety defect of that magnitude would clearly be visible in the data, and it's not.

So people invested in the argument now have to resort, like you just did, to theorizing about a literal conspiracy to hide the data that you know must be there even though it can't be measured.

It's just exhausting. They're fantastic cars. Get a friend to give you a ride.

Really? Wow. That's a lawsuit waiting to happen.

No way would I sign, and they'd fix it, or see me in court.

And not rich guy wins US court, but Canadian court. And yeah, it's different.

Don't you have to waive your right to sue in the US to purchase or boot a Tesla?

No, they have an arbitration clause, but you have the right to opt out.

Tesla leads the world in driving fatalities related to AP and FSD-type systems.

The entire rest of the industry has 1 fatality. Tesla has dozens, and 14 of those are old enough (and located in the right country) to be part of this investigation. (The multiple Tesla autopilot/FSD fatalities from 2022, including the 3 from last month, are not part of this investigation.)

The proper comparison is AP vs. other SAE Level 2 systems throughout the industry.

So, the rest of the industry has at most one fatality? How does that change the conclusion?

If disabling FSD makes teslas less safe then what is the point? Are they saying fsd can potentially go berserk? Are we into future crime prevention?

Yes, in the same way that taking down an active gunman with a loaded weapon is future crime prevention.

What about possible future software update to any hospital system? Should we preemptively stop all of those?

If the manufacturer isn't adequately testing for regressions that kill people, then yes, we should block those updates, and use the software the device was certified with.

gl getting anything done with a government agency defining adequacy of your software testing. here in canada government employees cannot get software that pays them salaries to work https://en.wikipedia.org/wiki/Phoenix_pay_system

Reply on news.ycombinator.com