i hope they keep that bug locked down. if that was weaponized there would be an issue for sure.
The most interesting part of this, is that this is a case that humans screw up on a pretty regular basis too. However we understand the human reason for doing so (lack of concentration), and understand that AIs don't suffer from the same fault. So when an AI screws it up we are still surprised.
This isn't a case where it makes any sense to try to extrapolate from training data and call it a day. You don't need any training to avoid hitting stationary objects.
Compare this comment from 2018: https://news.ycombinator.com/item?id=17015528
> The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. That's how the off-road DARPA Grand Challenge vehicles had to work. "Flat road" is a pure geometry thing. Not much AI needed.
Why use "AI" to solve a problem you don't have in the first place? This is the wrong tool for the wrong job.
(Compare also GPT-2 chess, in which a model trained on transcripts of chess games isn't able to stop itself from making illegal moves. The solution is straightforward - give your chess model a model of the rules of chess! And don't attempt to drive into spaces you cannot physically occupy.)
Is there a relevant difference between what's "training" for an CPU driver and what's training for a human driver, and if so, what?
Computers have problems humans don't too of course... that's why we get these articles about "ridiculous crash happened".
I repeat myself: "I can see arguments that the program shouldn't hit stationary arguments, but the ones I see apply to humans too with only minor changes." That humans have extra problems doesn't really matter, so long as humans require training to cope with those problems that are similar to those computers have.
For example, computers may use a vision device that sweeps across the surroundings and then use lots of postprocessing to build a complete scene. That's similar to what the human eye and brain do, except for the details of how the sweeping sweeps and how the postprocessing puts together a picture.
You do, because you need to correctly conclude that there is going to be a stationary object there with high probability. As it turns out, that's difficult to do perfectly.
You need to know that there is a stationary object in the first place, of a kind you care about (not a plastic bag). I.e. that it's not actually a moving object, an optical illusion, etc.
You also need to know that you are going to end up moving through it (it's not a parked car that you will avoid by following your lane).
That's... not trivial.
(going to be was probably poor phrasing on my part, sorry)
And "add some partial blockage by a vehicle in the lane, retrain the AI, and then redeploy it" isn't a proper answer... wouldn't that be a different model AI that would need to be validated again?
> And "add some partial blockage by a vehicle in the lane, retrain the AI, and then redeploy it" isn't a proper answer... wouldn't that be a different model AI that would need to be validated again?
You're the one adding unnecessary constraints here. If the validation process harms more than helps, you don't do it. In all likelyhood the correct validation process is something lightweight that can be done quickly for simple changes.
This sort of training process is likely to be a continuous process, they're always ingesting new data, trying to learn how to drive better from it (and keep up with how patterns in the world change), and therefore continually deploying new models. Because that makes driving safer overall. Not a "this bad thing happened so let's figure out how to put out a bugfix for it".
That is not true at all these days, not to mention that even a slight change in illumination will result in completely different input data, the way current AIs work.
In my opinion until this is somehow solved "full self driving" in presence of other human drivers sharing the road is just a dream.
> According to recent data taken from the AAA Foundation for Traffic Safety, approximately 12 percent of all interstate highway deaths result from shoulder accidents. This means that an estimated 600 people a year are killed—and thousands more are injured—while making emergency stops off the highway.
https://aaafoundation.org/wp-content/uploads/2017/12/Pedestr...
It's unfortunately fairly difficult to find stats on accidents scoped solely to vehicles rear ending emergency vehicles partially obstructing a travel lane. The stat I present is a proxy.
There is always risk. You can de-risk, but it's usually impossible de-risk entirely in regards to life safety. You are at risk of death on the road with human drivers, and that risk is well quantified (NHSTA, NTSB, private auto insurance industry). Arguably, there is data to demonstrate that safety systems and lane keeping are safer when edge cases don't occur [1]. The question is: are those edge cases acceptable if the risk is reduced overall? Planes crash, but people still fly commercial. Again, we don't ban commercial air travel, but regulators and industry work together to ensure there are robust systems in place to remove as much risk as possible.
Should we ban the production and sale of lettuce too because a few people get sick or die [2]? Because it sure does sound like we say, “oh well, caveat emptor, maybe start cooking your lettuce instead of eating it in salads." (The same number of people have died from lettuce as from Tesla's Autopilot in roughly the same window of time [a decade, lumping together AP1, AP2, AP2.5, and FSD beta])
[1] https://www.tesla.com/VehicleSafetyReport
[2] https://www.buzzfeednews.com/article/venessawong/the-decade-... ("2010s total: More than 1,010 people sickened, including 6 deaths.")
Not if they keep hitting parked cars they won't, the law will catch up to them.
How many deaths in total have occurred related to Autopilot use? 6 [1]. About 100 people per day die in auto-related incidents. If you want to take it to the extreme, Tesla has killed far fewer people than Boeing's 737MAX autopilot (which has killed 346 people in a six month span).
>> Not if they keep hitting parked cars they won't, the law will catch up to them.
> Uber killed a person with a safety driver not paying attention and all safety systems turned off. Did the law change? It did not.
Hmmm. You seem to have glossed over the verb aspect. How many people did Uber kill like that?
The link you posted shows a clear correlation between low light conditions and hitting people on the shoulder. A police cruiser has blue lights and presumable the federally mandated reflective surfaces that all vehicles sold in the US have (usually in the tail lights).
This is a typical case of software making stupid mistakes a human would never make.
Humans making this mistake do tend to brake at some point, just too late. There's still an accident, but damage is mitigated and survival odds, with seat belts and airbags, are good. Teslas are notorious for plowing into unrecognized stationary obstacles at full speed.
Many drivers don't get that limitation of Tesla's system. Which is why Level 2 systems that get it right most of the time are dangerous. Tesla has a really good lane keeping system, which creates the illusion that it can drive well. Until it can't.
> Distracted driving is dangerous, claiming 3,142 lives in 2019.
https://www.nhtsa.gov/risky-driving/distracted-driving
> The National Safety Council reports that cell phone use while driving leads to 1.6 million crashes each year. Nearly 390,000 injuries occur each year from accidents caused by texting while driving. 1 out of every 4 car accidents in the United States is caused by texting and driving.
https://www.nsc.org/road-safety/safety-topics/distracted-dri...
Is it too much to expect technology to at least avoid the kind of accidents only incapacitated humans have?
"Pedestrians may enter the Interstate intentionally, often despite restrictions and controls, while drivers and motor vehicle occupants may become “unintended” pedestrians when their vehicle is disabled by a crash or other incident(Johnson, 1997)."