Eventually we need the NTSB to certify updates before they are pushed out over the air. Similar to what the FAA does.
These OTA updates are not ok for large machinery and endagers not only the Tesla driver but all others on the road.
These OTA updates are not ok for large machinery and endagers not only the Tesla driver but all others on the road.
That's not true. There are many situations where the AP could make a sudden turn and kill you long before you had time to react even if you were paying attention. And in fact, the situation in the video seem to be not so far away from that.
You're begging the question [1]. The autopilot has full control authority over the steering wheel, so if it fails, nothing constrains it from making a sudden turn. If it is "just lane following" then it hasn't failed (yet).
You can test this yourself in a tesla by engaging cruise control, then hitting a turn signal. This would normally initiate an automatic lane change - but keep your hands tightly on the wheel as if you wanted to stay in the lane you're in. The wheel will attempt to turn, fail as you're preventing it from turning - and the AP disables.
Sure, if you react fast enough.
Let's not lose the plot here. The original claim was:
"The AP can only kill you if you're distracted/asleep at the wheel and not paying attention."
(Emphasis added.)
And that's not true. It can kill you, quite simply, by producing the wrong control input in a situation where the available recovery time is less than your reaction time.
If you doubt this, then I challenge you to drive a car where the autopilot is under my control. (It will have to be remote control because no fucking way am I willing to be in the car with you when we do this experiment.)
[1] https://www.nytimes.com/2018/03/19/technology/uber-driverles...
[2] https://www.popularmechanics.com/technology/infrastructure/a...
[3] https://www.aviationcv.com/aviation-blog/2019/shocking-facts...
The "attention checking" has a delay of a few seconds on it before it'll start warning you to grip the wheel. If your hands were on the wheel, and you were paying attention, there's no reaction time, since the erroneous control input would be overridden by your hands keeping you in your lane.
Put simply, if your grip on the wheel was loose enough to where the computer-generated move could physically move the wheel, you weren't in control of the vehicle and would be generating warnings.
>And if you need to be paying attention 100% of the time anyway, what's the point of having an autopilot?
The same reason cruise control is a thing on every modern car. It down on fatigue, which in turn, should improve safety and comfort. You're still required to be in control of your speed, but the vehicle manages keeping you at the set speed.
Granted, a ton of active research is going on trying to prevent these sorts of problems, but it definitely isn't a solved problem.
Basically, ask yourself this: how robust is this software? How are you measuring that?
https://www.youtube.com/watch?v=ZBaolsFyD9I
Here's autopilot following a lane straight into a concrete barrier.
https://www.youtube.com/watch?v=-2ml6sjk_8c
You can't assume that autopilot won't screw up lane following and swerve into a large obstacle. In those situations it's not as simple as making sure the lane ahead of you is clear, you might only have a split second warning between Autopilot going into "casual murder mode" and a collision.
It can "tentatively assist" you in killing yourself and potentially others, but we already know that from the disaster in Mountain View, where this exact same thing happened.
The problem with behavior changes in autopilot is that the car needs to react the same way every time.
If you're going around a corner on a road which you've driven on for years without issue, and the car all of a sudden does something unexpected, the panic and over-correction reaction that everyone instinctively has tends to cause more accidents than just holding your desired course does.
It doesn't matter if it's a skid, or your Tesla trying to accelerate you into a wall at 70 miles an hour. If the car does something the driver is not expecting it to do for any reason (from fatally defective software to ice on the ground) the driver performs sub-optimally as a result.
And since this is Tesla we're talking about (who bakes in features like your car not starting before you upgrade its software, which is another "feature" about these cars that's going to get someone killed sooner or later), I'm willing to bet that the car doesn't warn you that this might occur. It just works fine for 6 months, then an update gets pushed, the car tries to put you into a wall, and you cause an accident because you're trying to stop the car from killing you and didn't check your blind spot before that.
That is a UI and UX failure of the highest magnitude, and it is completely unacceptable, no matter how well it otherwise tends to work.
Nothing warning that behavior of autopilot might change.
I'd love for my VW to get updates OTA without taking it to the dealer. But, I don't want to receive those updates without knowledge that the update has been tested sufficiently (and given I don't trust the vendor, I'd like NTSB or similar government body to do this on my behalf).
You mean, similar to what FAA might do if it didn't allow manufucturer self-certification instead, right?
Also, wouldn't NHSTA, which does safety regulation and standards for autos, not NTSB, that does accident investigation, be the natural agency?
The timing of this is precarious (given the recent 737 Max allegations)
Disclaimer: I own a Model S
Of course it's also looking like they should have grounded the fleet after the first crash, given the history of the aircraft prior to its last flight.
It almost seems like a right-to-repair issue, where manufacturers are going out of their way to avoid documenting how their products actually work for fear of losing control over them or of disclosing details that a competitor or patent holder might find useful.
There definitely needs to be a strong regulatory response to that kind of behavior on the manufacturer's part when it comes to safety-critical updates, or even updates that might conceivably impact safety of life. Which, in the airplane business, is basically all of them.
I don't see why a more rigorous release process would slow progress down at all. All the iterations that lead to progress should be done on test vehicles, not customer vehicles.
"Move fast and break things" is a development model that should only be applied to low-importance, low-risk systems. Most software development work occurs on such systems, and I think that narrows the perspective of the software development community as a whole.
It's quite clear in this case that Tesla doesn't have an organisational structure that fulfills the requirements for functional safety.
It will if they run it through regression tests that Tesla doesn't seem to have the discipline to.