Tesla crash victim had complained about auto-pilot in same location
abc7news.com
abc7news.com
Who wrote this statement does not understand software. If there is some kind of Heisenbug, those numbers are too small to prove that the software is ok, and actually the guy that died reporting that there was an issue there, and later crashing there, looks like a very interesting hint about a potential software bug that should be investigated. The Tesla statement is the equivalent of "works in my laptop" at a bigger scale. Also consider that 85k times since 2015 means that potentially only a fraction of those trips were executed with the latest version of the software. Moreover the street layout may now be kinda different triggering some new condition.
Why did you eat a bagel instead of eggs this morning? Usually there isn't a scientific answer.
In the present, AI models are already so complicated that it seems hard to get reproducible diagnostic results, short of just saving every frame of data that the car's sensors pick up. And for video that seems rather prohibitive. Imagine collecting all sensor data for each of those 200 trips per day along the entire stretch of road.
>Catastrophic interference, also known as catastrophic forgetting, is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information.
Obviously you can set the RNG seed to be the same every time too, but even that only works if your system is wholly synchronous, which a car probably isn't.
Note that I doubt Monte Carlo methods are common in the autonomous vehicle space.
I wouldn’t be surprised if Monte Carlo techniques are useful for all manner of things related to ingesting sensor data on autonomous vehicles.
I'm not in the automotive space, but I'd be surprised if there were a viable self-driving car team not using Monte Carlo methods somewhere in the vehicle stack.
e.g. If an image (from the forward camera) returns the probability vector: {"stop_sign": .9, "green_light": .1} You stop the car 10/10 times.
Methinks there is far further to go than we realise - especially with the current air of hubris around the entire endeavour.
The original guys (McCarthy, minskey, etc) at least understand the gigantic complexity of the undertaking in so far as we understand how the mind works at all, and they never reduced things purely to classifiers. That’s like 0.1% of the job. The real researchers take philosophy of mind seriously — technologists want a quick and cheap solution that cannot exist.
And yet all this complexity and purpose may potentially be simply the product of putting a large collection of matter in a heat bath for a few billion years.
Who says that this process can't be repeated via brute force simulation? That the emergence of complexity and life under thermal disequilibrium + sufficient number of degrees of freedom is actually a basic fact of the universe?
I don't see any rule saying why, with sufficient brute force computational power, we won't be able to set up better and better simulations that will allow for the arise of true artificial intelligences.
It doesn't have to be. There are alternatives to that.
We think the brain has 100 billion neurons. Making those work is a very complex network, lots of chemical gradients, bio-electrical systems, cellular-level systems, and probably a host of other things we don’t know about. So it wouldn’t surprise me if for single neuron you’d need to model another billion things at minimum.
Now factor in all of the other input senses, and those 100B neurons have increased dramatically for I/O. Plus a multiplier to model everything going on inside a neuron. Then there’s also modeling the communication layers between, effects of myelin sheaths and all that.
So we’ve established that just to run a single “step in time” (whatever that means, as that’s a computing concept), we’re into probably billions of trillions of calculations.
Now what those model have to make sense and do something coherent. Every future step. So now you need some kind of meta model that is able to push this forward and continue in the right trajectory. Now complexity is dramatically even higher.
Now that we’ve somehow figured out how to model everything required for intelligence, we need to search that space through brute force, as you put it.
To me it seems like the time required will be astronomical — Sun might have blown up by then.
So why go down this route? Nature solved this in parallel relatively cheaply through billions of iterations and billions of permutations over tens of thousands of years.
Seems easier to simplify the problem down to more basic stuff that works well enough and iterate / combine systems. Either that or find a way to grow a “brain in a test tube” that you can hookup to a computer. There’s also always humans available at mass scale for pretty cheap across the globe — sophisticated intelligence built in.
Maybe you meant sapience or intelligence, but I wonder if people think computers won't be sentient (as in self aware) in our lifetime, even if they are still not able to match human cognition.
Ofcourse, this statement by itself doesn't absolve Tesla's software of any error. It's important to know the actual root cause before saying who/what is wrong or not, something which the HN community (and in general Internet/people) isn't good at.
That must be like half the bay area. I get that it's written as a token of pride for you, but maybe worth to take a step back?
> It probably means that the visualization system does understand such scenes and is able to navigate it.
I think there is a pretty good counter example proving that it doesn't.
As someone who works on software for one of the automakers, I also want to note that there seems to be a lot of focus on the software... but it's also very possible the problem was with the hardware: something in the vision or sensor system was incorrectly installed, defective, was calibrated incorrectly, lost calibration during use, etc.
I'm sure the software can always be improved, but if most cars are driving through this stretch without a problem and his always veers, that sounds like something unique to his vehicle which points to hardware in my opinion. In cars, the software can only ever be as good as the data it receives.
He owns the car/work in apple/drive the same road to work - for about half year (i.e. >100 times) and according to the article he has just about 7-10 veering incidents here. Like one or two per month.
Advantage being that if you can provide representative, simulated input then you can increase the training miles by orders of magnitude in the same amount of time, limited by computation rather than physical mileage.
Trying to use them to model real world scenarios would be useless in practice, due to the Ludic fallacy [0]. Real life is too complex to be modeled in any simulation.
As for modeling real life with simulations, the data exists for every type of accident a human has encountered. If you incorporate such into a simulation, plus randomly vary every free parameter, then your simulation will cover more scenarios than any human driver can possibly experience.
Thus, simulations should be able to help an autonomous vehicle outperform humans by a large margin, which is the only goal that matters.
The obvious failure was the highway safety mechanism not immediately being replaced after the previous crash - that would have made the Tesla crash have gone from death to perhaps only minor injuries.
I would argue too there is a need for AI image analysis - could perhaps be crowdsourced - to analyze the structure/state of a highway or any road. It would of course need to be trained, however it could also likely serve as tool to improve road safety worldwide - bringing everyone in line with known best practices. In this case, it would have caught the improper safety mechanism not being replaced, and then that death would have been avoided; cost-benefit analysis says this system would be worth it as life is invaluable.
If Tesla/Elon leads this effort it would show them being proactive in improving future safety, accountable, and taking ownership. Why has no other auto manufacturer done so, why has no government implemented such a system? Well, because honestly we are stuck in scarcity only caring about our own costs, and so if the random person dies here and there, there's not a big enough ripple effect (unfortunately) to cause enough personal worry to cause change. Tesla, if wanting to posture as being the steward I think many of us hope they are, does however absorb these ripples that get associated with their brand - and therefore the responsibility does pass onto them, whether they agree to accept it or not.
All auto manufacturers would likely benefit from this, allowing their software to work better within certain expected constraints. Many more possibilities as I think of it, newly spotted damage or cracks in bridges, detection of debris on highway or excessive dirt on the edge of the road making the conditions more dangerous for emergency stopping/maneuvering, etc.
Edit: I added a few sentences, so the one upvote prior to editing, they didn't upvote everything I said though it's all in line.
I fully agree with you here. Urban areas are too dynamic to rely on static maps and local LOS only. This sort of data sharing is going to be a necessity, IMO, to achieve level 4 or level 5 autonomy.
EDIT:
Yes he did and lost[0].
Jeremy Clarkson did another bit in The Grand Tour about the new Tesla and it was quite British[1].
[0]http://www.thedrive.com/sheetmetal/12536/remember-when-top-g...
There are over 200,000 cars a day driving at that stretch of highway, if a car crashes there once a week, then the Tesla Autopilot is an order of magnitude more dangerous than a human driver.
Edit: in a posting today, Tesla has said more: https://www.tesla.com/blog/update-last-week’s-accident but at the time of the earlier blog posting, Tesla says they did not know yet.
Let me share an example. I had an accident in a Model S, front-end, which didn't cause a loss of power. Tesla called me a couple of minutes later, saying that they knew I'd had an airbag-deploying accident, and asking if I needed an ambulance. From the description of this accident, power to the Tesla was lost before there was time to phone home.
Notice it says one fatality per "320 million miles in vehicles equipped with Autopilot hardware." But, how many of those miles (or what percent of the time) does a Tesla driver use Autopilot? Also, maybe those good stats are due to the structural safety features and not the autopilot. It may be more fair to compare to other cars with excellent crash protection but no autopilot.
I think it would be a great service to the world to improve driving safety, but maybe we need to really start looking at the stats and get some more hubris as we transition to full autopilot. For example, require that drivers keep their hands on the wheel and eyes on the road. When more cars have autopilot, then mesh behavior between vehicles, and e-road features, then maybe we'll be more ready for driver-less cars.
Also note that this includes trucks and motorcycles, which have much, much higher fatality rates than passenger cars. Motorcycles are around 10x-50x higher fatality rate than cars! So already Tesla's blog is doing a misleading comparison to more deadly vehicle classes.
Additionally, there could be all sorts of other variables that make it hide even comparison. Are Tesla drivers physically comparable to regular drivers? Are they older? Younger? Both elderly and young kids have a higher accident rates. Are Teslas driven in rural areas and urban areas the same as regular cars? Because rural areas have a higher fatality rate.
So we need to make sure that the Tesla driver matches driving conditions of typical cars to make autopilot comparisons valid.
Right now, it appears Tesla's autopilot is a death-trap.
I agree with all the holes you poked in their stats, but then with the last sentence you just went way off the deep end. What does "death trap" mean to you? To me, it seems likely the Autopilot engaged is about as dangerous as disengaged.
That's very curious wording. The important number would seem to be "millions of miles per fatality while Autopilot is enabled", not "millions of miles per fatality in Autopilot-equipped vehicles". The way it's phrased, I wonder what part of that 3.7x multiplier is unrelated to Autopilot and is just due to the car's high safety rating in a collision, recent release (no long tail of older, less safe models contributing to the statistics), and luxury status (more likely to be driven conservatively by old rich people.)
(Keep in mind that until they recover the logs, we don't even know if the autopilot was on.)
>In the moments before the collision, which occurred at 9:27 a.m. on Friday, March 23rd, Autopilot was engaged with the adaptive cruise control follow-distance set to minimum. The driver had received several visual and one audible hands-on warning earlier in the drive and the driver’s hands were not detected on the wheel for six seconds prior to the collision. The driver had about five seconds and 150 meters of unobstructed view of the concrete divider with the crushed crash attenuator, but the vehicle logs show that no action was taken.
So, clearly, the AP is not capable of avoiding such obstacles yet, even when unobstructed. It also seems like a mistake to not slow down to a survivable speed when hands are off...
Scenario 1: 100 human drivers drive some distance. 2 bad drivers caused 150 accidents and both died in their last fatal accidents.
Scenario 2: 100 self-driving/autopilot-enabled cars drive the same road for the same distance. Everyone has one accident and one of them is fatal.
Statistical numbers: Self-driving/autopilot-enabled cars cut both the accident rate and death rate by half. Death rate per accident is reduced from 2/150 to 1/100.
Conclusion: self-driving/autopilot is better.
"Tesla also posted these photos that raise another important question: they show what's called a "crash attenuator" or safety barrier in the proper condition ... and the way it was the day before Walter Huang's crash ... collapsed after a different accident."
Did you see any news coverage of the previous accident? I know I didn't.
It's a strange accident, especially because of the strangeness of tesla and the whole background story (broken safety barrier). I'm looking forward to the outcome of the whole story.
At the end I guess more than one party could've been a bad actor. Tesla, The guy itself and even that the road had no kind of warning (safety barrier not replaced or some kind of other warning sinces that it will be replaced soonish)
Something that will also be great will be a sort of "crash dump/bug report" button for these cars. If at any time your car does something unsafe, you can hit that button. The car will save the last 60 seconds so the manufacturer can analyze it to debug and figure out what went wrong.
I was so excited about auto-pilot and dreamed of getting in my car and sleeping while it made cross-country trips. So much for that, that seems way far out.
This is more a hijack than a direct critique but I think the general issue is the assumption that we need a private vehicle for that.
Trains are perfectly capable for bringing me from A to B while I am sleeping for years now.
Edit:// with years I mean years. Before that pressure made longer rides usually more uncomfortable than they are now with modern trains. Probably only because my area has plenty of mountain tho.
I think demand for trains has diminished which is why they're not as nice as they could be; However I found plenty of nice trains on offer in Europe and Asia.
For a very limited subset of points A and B, though...
Getting to, into and away from the airport easily ads another 2 hours tho. Also this route has 3 flights a week compared to a train every 2 hours.
With the train I just get in and enjoy nice service, free power and WiFi and usually get some work done.
Not to mention the difference in my ecological footprint.
My point is that it's not like you are wasting time in a uncomfortable environment usually. So I dont mind taking the train, actually usually even prefer it, so do millions of people all over the world.
Only as clean as the electricity is produced. In many countries electricity production relies on coal, petrol or gas, which are far from being clean energies.
In most parts of the country we don't have dedicated passenger tracks. Amtrak (our passenger line) just rents track time on the freight lines. The problem is that freight lines are often single track lines so if there is oncoming freight you just wait until the freight clears. Sometimes this his hours.
Until we have dedicated passenger lines, long distance train travel is a non-starter here. It was too slow for my 75 year old uncle on a site seeing tour. His exact words were "fun, but never again"
Tesla actually has that, it's not widely documented but you can do a voice command of "Report bug" followed by a brief audio description and it's uploaded to Tesla.
No idea how it gets triaged but I wish more products had a similar feature.
Seems an obvious way to improve: look at where your drivers are overriding your system.
Good point, and it isn't just vehicles with auto-pilot perhaps all vehicles.
The 2017 and 2018 Chrysler Pacifica Minivan kept on randomly shutting off[0] and loosing all electrical power at freeway speeds. You had to put the vehicle back into park, on the freeway, to restart it. There are several stories of near death experiences trying to get to the shoulder. It was intermittent.
This is relevant because nobody could figure out why, even the manufacturer. They had to install custom self-powered diagnostic equipment in consumer's vehicles to even try and get enough data on it. Turns out that suddenly cutting power to the onboard computers doesn't make them very useful for diagnostic analysis, some dealers even acted like the problem didn't exist because the computer had no record.
They did ultimately get it fixed (replaced an engine management unit or similar) but it does go to show that modern vehicles are getting so complicated that even manufacturers are struggling to stay ahead of it. Tesla's autopilot and similar just takes that up another notch.
[0] https://www.autoblog.com/2017/11/21/chrysler-pacifica-owners...
A lot of that is for the infotainment systems, but regardless, we're probably going to be seeing automakers struggle more and more with all of the problems that have plagued the software industry for its entire history.
On the one hand, I want to say that situations like this Tesla crash and your minivan example get so much attention because that's rare, so the manufacturers are mostly getting it right and we'll all be fine. On the other, knowing what I know of the software industry plus my past experience as a shadetree, the prospect of that much code running in 65-mile-an-hour murder machines gives me the shivers.
It's sort of natural - it's "why not". We can afford to have more SLOC than ever before. Why bother to try to reduce it, that's not a priority, our priorities are features and fixes. It becomes too much to really understand and manage, and then it's all automated checks and automated tests.
It's very hard to get the average/common developer to spend time thinking about how we can remove stuff, clean stuff up. And managers don't want to pay them to do that. In some places it eventually becomes a big problem, and then it's time for a big rewrite. Most software is throw-away. Website front-ends, mobile device drivers, etc ... mostly thrown-away roughly every year. It's the relatively rare and stodgy classic open-source stuff that's re-used (with minor updates) for decades. OK, I'm getting off-topic ...
I'm not sure what "verifiable models" means here. There are a bunch of people using Simulink to "auto-generate" code for some ECUs. I've used it with great results, and I've seen others do that as well. And yet some people still create horrible things with it just like regular code. But not once did I see anything that sounded like formal verification. That doesn't mean it never happens but the way The Mathworks markets their tools, you'd think some kind of magic is done automatically.
On a related note, I always laugh when Mathworks promotes the ability to compare "simulation" results to that of the "generated code". They call it "software in the loop" testing. This is actually getting their customers to verify that the tools they provide work as advertised.
For that kind of development I'd actually prefer to just shut the simulator off always generate code. Simulation should only ever be for plant models, not control systems you want to actually implement in software.
To me formal verification has always meant proving correctness in a mathematical sense against some sort of formal specification.
None of that really matters though because AFAIK you can't prove a neural net has been trained "correctly" and doesn't contain errors.
Progress is being made in this area. Here is an "old" paper: https://arxiv.org/abs/1702.01135
My only experience of automated driving is the feature where you can set a maximum speed. This is useful in the UK since average speed cameras are commonplace on the motorway network nowadays and it is easy for your concentration to lapse and drift over the current limit. The cameras are automated and there isn't much leeway.
However, the speed limits often change in range of 50, 60, 70 mph depending on local conditions and roadworks, so you can never just 'set and forget' you need to be alert all of the time since speed limits can change frequently.
It's going to be a long time before I trust a car to do this on its own without my supervision and, as regards going to sleep and letting the car do 'everything' on its own, I don't envisage being able to do this comfortably in my lifetime.
Update: Since Oct'16 self-driving software is an add-on option and Musk has been tweeting[1] since Jan'17 that it's just months away and that its autopilot is somehow safer than adaptive cruise control. This Oct'16 video even starts with the claim that the driver is only there for legal reasons.
There is a lawsuit since Apr'17 regarding above - Google dean sheikh et al v. tesla, inc
[1] https://twitter.com/elonmusk/status/823632597284691969 " Yes, safety should improve significantly due to autonomy features, even if regs disallow no driver present 23 Jan 2017"
3 months maybe, 6 months definitely 23 Jan 2017
They are not careful at all. The first text on that page says "Full Self-Driving Hardware on All Cars"
You have to be moderately savvy to realize that you'd also need this to be coupled with "self-driving software", which the Tesla does not have.
By doing this they are in my opinion sabotaging the invoations they are actually making.
When I say that it's at least ten years away and likely much longer because it will take big breakthroughs (strong AI, lot's of knowledge about the world, etc.) they look at me like I'm crazy.
I really hope it happens in my lifetime, but I remember talking about Greenblatt's chess program (see [1]) when I was at MIT in the 60's. The AI folks really thought that a chess program would be world champion within 10 years. They finally did it, but it took more like 30 years, and one of the reasons they were successful was that Moore's law eventually allowed a super computer (Deep Blue, at the time the 259th most powerful computer in the world) to evaluate 200 Million board positions per second[2]. Deep Blue's opponent, world champion Garry Kasparov) was most definitely not evaluating 200 Million board positions per second.
I see driving as a much more complex task than playing chess, so maybe 30 years before we will put our children in cars to be driven across town. I will be happily surprised if it arrives sooner.
[1] https://dl.acm.org/citation.cfm?id=1465715
[2] https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)
It may take 30 years to handle blizzards and every edge case but you can just park the car to avoid those problems temporarily while still serving 99% of the market so they don't need to be solved now.
I wonder if we would have had safe planes if that was the mentality when they were first invented.
If it's any consolation that's not how autopilot on an airplane works. Pilots are expected to remain alert and ready to intervene in case of trouble.
For babies car is autonomous already. They have to be strapped pretty tight in a bit reclined position.
Do you mean to sleep sitting just like passengers in a car kinda can? I guess people are able to sleep that way. I for one have to be totally exhausted to fall a sleep sitting.
One certainly can't hope to sleep laying on a side under a blanket. Self driving can't ensure that car will never experience sudden deceleration. For the same reason you will not see seats facing back of the car. Because you could be killed by your or other passenger's laptop or burned by your hot coffee.
Ideally, this would be sent to the NTSB as well to determine if they need to issue a recall and possibly also shared with the public if the car's owner is okay with that.
Issuing a bug report to the manufacturer is fine if the manufacturer is being responsive, but fixes take time and large companies don't always put a high priority on fixing bugs. Sharing bug information more widely increases their accountability and I think potential buyers have a right to know about safety-critical flaws before they buy a car.
Perfect self-driving cars is a nearly impossible feat to accomplish in an unbounded track. I can only imagine automated driving in a system which has no room for error. Examples include: tunnels under the ground, chain links on the ground (as in trolleys, trains, etc.), or anything else that vastly reduces the entropy involved in driving.
With self-driving cars on current roads, it will probably take years to get from 1% error to .1% error, and decades to get from .1% error to .01% error, which isn't even good enough. Perhaps it will take a century or longer to develop the required artificial intelligence to make self-driving cars perfect "enough". There's just too much room for unique problems to spawn. Bounding vehicle freedom seems to be the only way forward.
Furthermore, look at the actual data we have right now. SDC makers actually put out data in California about their "disengagement rate" which is how many time the human drivers took over from the software. Waymo have steadily increased that rate over the past few years, now they are driving many hours without disengagements. Look at the link below, page 4, you'll see they have 63 disengagements over 350k miles. That's 1 per 5.5k miles, so these cars are driving for days without a human takeover.
They will not need their own infrastructure, that would be not be economically viable. They will go on the roads we have or they won't go at all. Tunnels are going to be reserved for high-density point-to-point travel, if the boring company or others ever get scale...
Put differently, I should be able to lay down for a nap in the back seat and wake up at my destination without any chance for disengagement during my entire lifetime. At the current rate of 1 mishap per 5,500miles, I would be dead after about 6 months.
Assuming a human lives to 75 years (we should really be using 75years minus 16years, but it's unimportant), a lifetime of driving is about 1,000mi/mo x 12mo/yr x 75yr = 900,000 miles. I don't even want the probability of encountering a mishap to be once per lifetime, let alone once per 6 months. One mishap per 900,000 miles isn't enough, because, on average, I'd encounter one disengagement in my lifetime. Assuming we're striving for a world where 7 billion people can drive without a single incident in 75 years (a vast underestimate), we need the probability of a mishap to occur to be less than once per 7,000,000,000humans x 900,000mi/human = 63 x 10^14 miles.
1/5e3 is not even close to 1/6e15. We're talking about 12 orders of magnitude in our error rate. I'd say we're laughably far away from our goal. We've got a long way to go.
[0] https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115...
[1] https://www.fool.com/investing/general/2015/01/25/the-averag...
This doesn't seem reasonable - Waymo's report doesn't dive in depth enough about each disengagement to warrant this sort of extreme reliability.
If "2 disengagements" per year, were at most fenderbenders - something I'd wager humans do way more than twice per year - that would be a very different story than if those 2 disengagements were life threatening. Sure you'd wake up from your nap, but you wouldn't be dead, and at most you'd have to exchange insurance information.
So, you're right, there's no clear distinction, but I would further argue that it doesn't matter. Even if only 1/1000 disengagements are fatal, my conclusion remains the same. I think we're splitting hairs at this point, though.
Even if not fatal, I highly doubt a significant fraction of such events (as defined above) would allow me take a nap upon departure and wake up at my destination, so it would still be unacceptable to me. I guess we have to agree on what an acceptable end-game is for fully autonomous vehicles. If you think "waking up on the shoulder exchanging insurance" is acceptable, then that would indeed change the numbers (but by how much? Two, maybe four orders of magnitude?).
Humans get into fender-benders all the time, but surely we'd strive to eradicate this inefficiency in the automated driver. I think this is still an active area of debate; some assembly-line work can be made more efficient with machines, but we've seen humans out-perform machines in other types of work. I think driving tends to utilize more reactive, intuitive "System 1" thinking[0], so I imagine that humans will be vastly better than machines at driving for a very long time.
And no, it's not acceptable by any stretch. A disengage is extremely dangerous to drivers who are lulled into false sense of security through the marketing of.. self-driving cars.
I'd like to know what the actual rate would be, the 5k miles figure is for conservative thresholds of when control should be handed over, and many problems may be safely solvable (if annoying) by simply coming to a halt or pulling over.
No, the current setup is not OK to release on a large scale, but it's not expected to be, and we're not 12 orders of magnitude from a reasonable point.
And the old striping is lightly visible as well.
"[...] when it entered and traveled in an unmarked gore area, rather than the intended high-occupancy-vehicle (HOV) lane, and collided with a crash attenuator. The 990-foot-long gore, with an unmarked inside area, separates the left exit HOV lane for State Route 85 from the US-101 HOV lane."
The type of accident does seems comparable though.
Looking at the image and estimating the traffic flowing, it's simply amazing that CalTrans is allowed to take a week to fix such things. How long does it take to put up some sand barrels? This situation was made more dangerous because while many such barriers are used in freeway exits, the left lane speeds can be much higher. 7 days + weather is a long time to fix (some states require 3 days).
I get the feeling that Tesla/Waymo and other SDC manufacturers will be more proactive in reporting unsafe conditions as a priority to fix. Such a partnership could improve safety for everyone.
[1] http://abc7news.com/automotive/exclusive-i-team-investigates...
It's like they painted traffic lines to guide vehicles into barriers, and then didn't replace the attenuator after a human driver made the same mistake as the Tesla autopilot did a week later.
https://techcrunch.com/wp-content/uploads/2018/03/screen-sho...
Somewhere between 7-10 iterations of the same dangerous failure mode, it's time to stop betting your life that the failure won't repeat.
> Walter complained "7-10 times the car would swivel toward that same exact barrier during auto-pilot."
This still doesn't diminish Tesla's obligation to make their autopilot system behave properly.
But he wasn't operating it.
Of course people may have abused the system if it had a more sensible name anyway. But names competitors use like Automatic Cruise Control or SuperCruise seem much more descriptive of the system’s actual abilities.
The expectation that the driver will jump in to prevent a car's autopilot from steering the car off the road into an obstruction is ridiculous. It's one thing to expect the driver to stop for on-the-actual-road dangers. This wasn't.
We don't really know what happened yet, but if it was the autopilot and it did steer into the barrier, this is at least 95% Tesla's fault.
As we see in latest Tesla report, there were multiple warnings to which the driver should respond: this means, that the problem was detected with enough time to react and the autopilot was not able to solve it on its own. Thus, it does not make sense to blame the autopilot - the technology did behave as expected (when problem cannot be solved, report asap to the owner, that he should intervene).
What Tesla can actually be blamed for, is the misunderstanding of the autopilot capabilities by the drivers. It's not an algorithmic bug, but rather a usability one. Perhaps, the warning should be issued earlier or in a more clear way. Perhaps, the drivers should receive some training, to act as supervisors of the system, not as passengers, or there even should exist a special driving license for such types of cars.
Nevertheless, the main fault was not the driver's and not the Tesla's. The safety of this road should be guaranteed by the authorities in charge, the navigation should be made sufficient for the drivers, and they clearly failed to do so.
Seriously though....I doubt it. The report said he was told to put his hands back on the wheel "earlier in the drive". I seriously doubt he would have had time to react in this case. This is not the same as a plane...there was no "ground alert" or anything like it. From what I can tell, it was doing its job then steered right into a wall. That sort of thing simply doesn't happen with a plane....when autopilot is active on a plane, there is never a situation that requires instantly noticing something is wrong, and correcting it, within a few seconds. Never.
And usability bugs, if that is what you want to call this, are indeed Tesla's fault. They, not the driver, have the resources to test and understand how human attention works (i.e. "vigilance deficit" / "handoff problem")
In a commercial jet - Goes to the toilet? Eats lunch? Chats to the co-pilot?
Sure none of those might be right but I'd warrant that a lot of people would answer with things that move the pilots concentration away from piloting, "because the auto-pilot is doing the flying". Honestly I'd assume they do any of these answers; but with the proviso that the co-pilot stays in the cockpit when the pilot leaves.
Would it make a difference if you learned that the driver knew there were problems with autopilot and was using it anyway?
Tesla should improve its systems regardless of whether it was the first such error, the seventh, or the tenth.
But mostly... I just don’t understand it. It seems like such an odd decision by the driver I don’t know what to make of it.
It’s very unfortunate that he died, but that doesn’t mean we can’t ask questions/criticize what may have gone on.
That may be what you extracted from this incident, but it's definitely not "the lesson".
If they only tried at the dealership and not the actual place... that seems like a big mistake on their part.
I'm a biker. I can drive, but only ever drive rentals; motorcycles are my primary mode.
A sense of righteousness for being in the right on a bike will get you killed. You ride within the margin of not just your own error, but also those of other road users, and your own machine, with your best judgement and risk preference tradeoff.
If your machine has a known tendency to do something bad in some situation, you avoid that situation, even if you have someone else to blame. Blame doesn't keep you alive.
But Tesla and everyone else says the owner is still ultimately responsible because AutoPilot isn’t a 100% situation.
If my car did something funky like randomly accelerate or turn hard at a given intersection and the manufacturer refused to fix it I’d stop driving through there.
Why push your luck?
I truly don’t like blaming the victim. I’ve been very critical of Tesla lately for their claims and safety issues.
But I don’t understand this man’s decision at all. It doesn’t seem reasonable.
I wish I knew why he kept using the system in that area.
The ONLY idea I can think of is when the car got an update he would try again to see if it was fixed and of the tries was sadly the last.
As for blame, the check engine light is for emissions gear, and will not effect safe vehicle operation. A wobbly front end, however, can indicate a serious problem, and yes, it is the driver's responsibility to operate a safe vehicle on public roads, and it would in fact be their fault if something happened because they did not perform proper maintenance.
I'd get this fixed, if I'm right you're going to lose half the braking power of your car soon. (Pretty much all cars have a dual braking system, if shit happens you lose one front and one rear brake but not everything)
I don't even think that even applies in this case.
If TFA is correct and there is a problem in this particular spot and you still rely on autopilot then who is left to blame except the person who entrusted their life to a system known to be faulty?
It sucks that the driver died, and it sucks that the Tesla autopilot system had problems handing that kind of situation, but that does not mean the driver is blameless. He put himself and the people in the cars around him at risk by using the autopilot feature on a stretch of road where he knew it did not work well.
1.https://www.ntsb.gov/news/press-releases/Pages/NTSB_releases...
Because the driver didn't receive any training from the manufacturer. Airplane pilots, in contrast, receive a ton of training right down to how to fly a specific type of aircraft(single-engine, twin-engine, instrument flying, etc.). Additionally the manufacturer will provide training to pilots on how to operate any nifty features of the commercial aircraft.
I don't believe Tesla provides any training whatsoever on how to use these features. And I'm not aware of any mechanisms preventing untrained users from activating these features. A tutorial that you can click through does not count because you do not ensure rapport with the trainee like you would in a person-to-person training.
Back when cars were being commoditized the dealer would often provide training to new drivers. And in all states new drivers are required to take a practical test to demonstrate that they are competent to drive. Does Tesla require their users to prove any sort of understanding or competence before they unlock Autopilot?
You might argue that requiring training sets a dangerous precedent, but users need to be made aware that the driver assistance systems are not foolproof, and the only foolproof way to do that is to require them to attend a training.
Tesla is beyond irresponsible with this and IMO they should be sued out of existence.
It isn't a new feature on a cell phone that you just watch a youtube video on and move on with your day.
They should have been able to fly manually and safely land with a faulty radar altimeter. It is likely the crew didn't understand the significance of the fault, even though Boeing had issued previous warnings.
[0] https://en.wikipedia.org/wiki/Turkish_Airlines_Flight_1951
"He was right, dead right, as he drove along, but he's just as dead as if he were wrong."
Maybe I’m old fashioned, but I expect all of the technology in my mass-produced car to be safe, where safe is defined as “will not steer my vehicle into a barrier at freeway speeds.”
I don't understand most people's perspective on this. It's like walking out in a crosswalk without looking; sure if you get hit the driver may go to jail, but you'll still be dead (or seriously injured).
Use your own brain, you can't rely on others to keep you alive. At the end of the day you're the one with whose got the most to lose, always.
> Let the autopilot park your car, if it makes a mistake the only bad thing that can happen is just a few scratches
Or it can drive over a toddler or pin someone against a wall until they die.
There might be situations where this could happen but humans have such a bad track record when it comes to driving/parking/paying attention in general that i'd trust a good autopilot over the average person. There will be bugs but at least we're able to work on them. Improving human driving skills seems unfeasible with any other current technology.
I’m not saying this is a winning argument, especially since he did so much to bring it to their attention, but just that Tesla would be expected to raise it.
Of course, they’ll probably try to settle this quietly and hope that everyone goes back to worrying about the Uber self-driving fatality...
I mean, the crash attenuator had recently been used up, but the dangerous line situation that led to the crash which destroyed the attenuator had not been fixed.
"Man you have got to be ready — it makes mistakes, it loses track of the lane lines. You have to be on your toes all the time," says Wozniak. "All Tesla did is say, 'It is beta so we are not responsible. It doesn't necessarily work, so you have to be in control.'
"Well you that is kinda a cheap way out of it."
But I took a test drive in a model S for the first time earlier this year and almost immediately noticed autopilot’s extremely unreliable behavior— it would swerve out of lanes in ordinary situations that should have been easy to handle. The second I saw that, hat was it: I would never use it again before many years of testing and improvement had taken place. No way am I gambling my life on a clearly incomplete feature just because it’s cool. Fuck that.
Of course Tesla is fairly safe behind their disclaimers and warnings, and to be honest I think it may be impossible to develop such a system without putting it into the wild before it’s perfect.
But for me, personally...I’ll let other people choose to be the Guinea pigs. The risks are all too obvious. Continuing to use the feature is very dangerous. Do it knowing this may very well happen to you.
I still wouldn't trust it. I don't believe their current hardware has enough processing power to do the job. But it should be performing better now than even a few months ago.
I don't think so. All it will take is a Tesla plowing into a sidewalk full of people, and no hand-wavy gesture at a license agreement is going to make the political and legal pressure go away.
Wow, that attitude strikes me as a little creepy. If you admire Musk to the point of hero worship, that's perfectly fine. But why worship a corporation?
I've seen something that said that the car was warning him to take control, but he hadn't done so. Texting, maybe?
>The driver had received several visual and one audible hands-on warning earlier in the drive and the driver’s hands were not detected on the wheel for six seconds prior to the collision
Note the and.
Further more.. if the barrier was already hit previously by another car, as the reason why it was collapsed prior to the Tesla crash, this may simply be an area where the road is not designed very well and causes many human error in that location. This happens all the time. In fact, road design is a significant cause of vehicle crashes.
It sounds like they are saying the driver had 5 seconds to notice the problem and react. It is scary to think that when you are on autopilot you may be 5 seconds away from death at any time. Better not take those eyes off the road!
I had my driver side view mirror stuck issue fixed and they put in the notes that they re-filled all my tires during repair service allegedly as reqd. by some CA law. Later that week I got an alert that my tire pressure was low and had to get it air pumped so obviously they didn't do that despite having claimed to have done just that...
This is how Tesla reacts to all bad publicity. Bad review on battery life comes out? Data dump indicating the reviewer might've done something wrong. Accident? Statistical dump showing that most cars don't do this. I like this technique more than generic "we can't comment" responses, but I'm pretty confident they heavily cherry pick, looking for something that people who glom onto data can see and say "oh good, Tesla's right."
Why not get a mechanic to ride along with him to that location, perhaps with extra diagnostic equipment connected? I could see how a bug that is dependent upon being at that location would certainly not be reproducible somewhere else.
Here's an automotive service booklet from almost 70 years ago which recommends the same thing for troubleshooting:
http://www.imperialclub.com/Repair/Lit/Master/021/Page14.htm
...and after 7 to 10 times , he still didn't learn his lesson? That's pretty stupid if you ask me. If my car does something weird at a particular stretch of road, especially 7 to 10 times, you can bet your bananas that i'll be paying a lot of attention on that stretch of road. If my "autopilot" (seriously, Tesla should stop using that name) isn't reliable in certain circumstances or places, then - guess what? - I WONT BE USING IT THERE. Why blame Tesla (I'm no Tesla fan), when the operator of the vehicle refused to operate it properly in the face of prior experience? Poor guy, and I feel for his family, but come on, what a dumbass
It seems like the Tesla autopilot is very similar in capabilities to other manufacturers' active lane keeping, adaptive cruise control and active collision avoidance braking systems, however the marketing and user behavior is much different.
There's no expectation that a Pacifica with all the bells and whistles is going to do a good job driving for you with no hands, but if somebody stops suddenly, it will too.
I think deploying self driving cars at <=25mph speeds at first would be wise. Personally, I wouldn't risk letting a car take over at high speeds until there is a longer track record of safety.
Even a small part of a sign should be enough. They're designed to be easy to spot.
It seems like we just set neural networks up to recognise all objects and assume they'll recognise simple objects too. However typically how humans learn is through simple cartoons first and then layer on top of that rather than the other way round.
Edit: should have done some googling before opening my mouth [0] [1]
[0]: https://amundtveit.com/2017/07/13/traffic-sign-detection-wit...
[1]: http://www.bartlab.org/Dr.%20Jackrit's%20Papers/ney/1.TRAFFI...
It seems all these companies are in a big rush and being slapdash. Maybe it's a disconnect between the engineers and the execs/shareholders. It doesn't even seem profit-driven...almost fear driven. ("we don't wanna be left behind.")
You can't just throw your hands on the air and hope for the best. Your car, you are on the wheel, you are responsible.
On a highway I think it even detects crashes ahead of you ( had once two cars in a three lane almost crashing while one car tried to switch Lanes)
Well, back to story. I have a street where cars park left and right of the street like a zick Zack. There is one spot where it always warns me about a crash..
They didn’t call it “self-driving” or “autonomous” because it isn’t those things.
A better analogy would be cruise control. It controls essentially one variable. As does lane keeping. You combine a couple of these things and you think it's smart, and it isn't. We learnt this (edge cases between single variable trackers) ~40 years ago in aircraft, that there are places in the flight envelope that combinations of single variable trackers will still let you go, but will also kill you.
Even with airplanes, if something gets in the way, autopilot won't always save you and can only alert you when the situation gets bad i.e. oncoming airplane or obstacle, altitude, speed etc. We fly in planes with autopilot and we are safer for it, but pilots/drivers still need to be alert and operating the plane. Teslas aren't fully autonomous and probably can't truly be until everything is connected and more cars on the road are autonomous for expected behavior. I trust autopilot in planes but still want a pilot. Most likely autopilot will be more useful in large buses, trucks, shipping, boats, airliners than individuals as you will still need a driver most of the time. Even the Uber crash could have possibly been avoided with a more alert driver.
One area that may cause more crashes in the interim is trusting the software too much, autopilot that does work six sigma 99.999999% of the time may lead to a possible issue of driver comfort with the technology that may still have edge cases that can endanger them. This issue was a factor in the Tesla crash and the Uber crash
I think a huge overlooked part of the failure here was the previous accidents at this part of the road and the lack of repair. We have a serious infrastructure issue with disrepair, non automated/untracked driving of humans probably ends up badly in these areas but goes unnoticed or is suppressed as road design can be a big factor, this automated crash highlighted an issue here at this offramp/fork that probably would go unnoticed and cause more issues. If nothing else, automated driving will have the data to fix these bad areas of our infrastructure and double down on safety and protections.
(Full disclosure: Their incompetence almost killed me a few times back in my I-880 commuting days)
Source? ABC7 says the previous crash was 11 days ago (DUI...), the barrier just wasn't fixed immediately.
http://abc7news.com/automotive/exclusive-i-team-investigates...
There is zero information yet and its all everyone is jumping onto. Last week the same thing happened to a Tesla that wasn't even equipped with autopilot.
Tesla is mincing words.
If I may: do you realize that, in this catastrophic wreck, that the fire only started, slowly, after the driver had been taken by EMS?
One big difference between a fire with batteries and gas is this: if gas starts to burn, most likely all of it is going to burn, quickly and often explosively.
None of these are the case with the batteries in Teslas. Only the damaged ones are likely to burn, slowly, and not explosively.
There should be manslaughter charges for this kind of thing.