Somerset cattle grid mistaken for wall by car sensors
bbc.co.uk
bbc.co.uk
But pragmatically, they likely don't answer the phone. Or email.
So, it is still easier to fix the road. And hope the issue doesn't resurface elsewhere and inadvertantly injure or kill too many people.
The solution is having the balls to lay down good policy/regulation. There is no point forcing car companies to use a certain algorithm, for example, but it's not unreasonable to build a standard basket of problematic road signs which the car must pass - e.g. a cattle grid isn't too bad, but if a driverless car doesn't stop for a speed limit outside a school the company should have some liability.
You don't get to abdicate responsibility on the road just because you bought in to some corporations marketing.
(While, sadly, at the same time it seems from the outside that the ACAS code got written by the same outsourced dev team that work on their website the rest of the time. :sigh: )
Imagine you’re driving along in full control on an empty country road. Out of nowhere your car panics, slams in the brakes, deploys seatbelt tensioners, and flashes bright warning lights because clearly you fell asleep or something if you cant see that big wall.
Would you react? Maybe swerve coz you’re startled?
Words mean things. You don't get to say they mean different things when you say them just because it helps you sell more cars. Not in the fine print, and not in a court.
If your company's automated emergency braking, or lane assist, or adaptive cruise control - causes or contributes to a problem, someone from that company should be in court next to the driver facing judgement for the consequences.
(I'd even go so far as to suggest if the existence of a system leads to a situation where an incident occurred because of the driver's assumption that the automated system would prevent it, then the company should need to justify why the driver should be held responsible instead of the software. If your advertising claims "autopilot!", you're going to be apportioned a significant portion of liability even if your fine print says "Drivers must keep alert and ready to take control at all times" - you should be judged based on a reasonable persons expectation from the language you use to promote your vehicles, not some w4easel words in your T&Cs.)
When you say that a manufacturer should be held liable if their system '... contributes to a problem...', it seems like you're suggesting adoption of the (notoriously problematic) 'if not but for' test. This sort of test is used in negligence cases, but can be useless, as almost anything satisfies it (i.e. slow service at a coffee shop may have been a necessary condition for the timing of that exact accident).
In practice, putting the responsibilty on auto manufacturers for every specific injury or death that happens due to a safety system, without somehow giving them "positive credit" for lives saved, massively delays or entirely prevents these sort of advancements from being made. We definitely need processes to verify that new features are actually functional, typically through recalls and product liability suits, but simply putting blame on manufacturers for all accidents is a terrible idea.
This is a particular challenge in California because our school zone signs specify that the limit only applies "when children are present."
Since a large fraction of kids are driven to school (at least in southern California), you know that this is happening because of the massive traffic jam near the school.
I'm wondering how you would get around metaparameter optimization here, or the fact that it's too easy to optimize to a known test procedure rather than the real world?
> So, it is still easier to
... hold them responsible in court for damage/injury/deaths caused by the actions their vehicles take which are not under control of the driver.
If your corporation (via it's self driving software) kills someone, the corporation will "go to jail", a C level exec does the jailtime and the entire company will be unable to carry on any of it's normal business for the duration of the sentence (while still being held responsible for it's obligations to staff/suppliers/cerditors). If that self driving software is judged to be liable for a traffic infringement, the corporation will pay an amount equal to the percentage of income that a human driver would have paid. If it's an infringement with a ~$400 fine, the corporation should pay ~0.5% of their annual income - the same sort of penalty as "an average salary" person.
A couple of $20mil fines and a 6 month total company shutdown would be very "proactive" in encouraging Tesla to get Elon to shut the fuck up with his big bold claims that cannot be backed up. If Uber had been slapped with the "corporate personhood" equivalent of whatever penalty a human driver who'd killed that person crossing the road with a bicycle they'd be far more circumspect about using minimum wage contractors as "safety drivers" in their prototypes.
(I can dream, right?)
I can see a wide spectrum between my (admittedly over punitive towards corporations) ideas and this (somewhat intentional misinterpreted) idea of yours. But I do honestly think Uber and Tesla have evaded morally deserved punishments by blaming customers or contract staff who were pretty much doing what everybody would expect to be able to do using the technology as described.
"it's inevitable that even a responsible _person_ will screw something up" - but that doesn't mean we give them a zero consequence break if they kill or injure someone. (We do, however, distinguish between things like murder, manslaughter, and negligence-causing-death. And penalise differently, based at least partly on the ability of the punishment to discourage other people from doing similar things. Perhaps Uber was only guilty of negligence-causing-death for using minimum wage contractors as safety drivers. Perhaps Tesla was guilty of manslaughter by advertising as "autopilot" a thing that drives full speed into offramp dividers?)
In the large, it's always a shame to fix a good test case.
This is the kind of issue which will persist, vehicles which fail to recognize a real world situation shouldn’t get the benefit of changing the world to fit them.
An inquiry should be done into how these vehicles came to fail to recognize this road feature and the upstream cause (not just the bug by the systematic issues in the organizations which produced vehicles with this bug) should be fixed before they’re rectified to operate on roads.
The good ones would then sort out why their braking system is faulty. Perhaps it gets confused about other things in other serious ways.
Do you really regard braking in front of a sudden incline/decline as ridiculous?
From uphill: "There's something weird in front of the car, best drive very slowly". From downhill: "WTF I can't see the road any more, best drive very slowly." Both seem quite sensible to me, I don't understand why so many people on the thread seem to disagree.
These grids usually lie flat on the ground and in some areas are commonplace. Seems to me these sensors are still needing work. Perhaps they see horizontal lines and panic?
Cars running off the road for no good reason is ridiculous. It is indefensible to assert this is OK. Denying it or otherwise covering it up is definiteky ridiculous.
I don't believe we'll solve autonomous driving too soon without adapting the infrastructure to it rather than the other way around. At this rate we'll have truly self driving cars (not "straight line, follow the highway lane self driving") right around the time everyone starts taking most personal cars off the road to replace them with something more efficient that better suits cities of millions. But right now too many people and manufacturers are lying to themselves that self driving cars are right around the corner, and even that they're mostly here.
The solution would be to fix the disinformation surrounding self driving tech and fining the hell out of any manufacturer that doesn't make it painfully clear on every channel that the tech is very limited, mentioning also the limits of the system (e.g. types of traffic it was tested on, how many road signs it can recognize out of the total, how much of the regular driving experience do those conditions cover, how well did it do compared to an average human driver). All in "human" language.
As someone who only sells products using ML, I find the discussion here inorganic.
Maybe most of the blame.
But consider this: If a human had the same problem, who would be blamed? If several humans, then the road would be changed, just like this. The process seems sound, but maybe the changes that make it more human-friendly make it more machine-hostile.
New thinking is needed on how we retrain machines after the change, as it necessitates uniformity - the machine would benefit with fewer variations in cattle grids for example. I reckon a simple change to the road markings for all cattle grids would be enough to make them logically identical - be it paint or signs or some other physical indicator.
Though I don't think fully autonomous driving will be possible in near future. There are too many edge cases.
I don't know that this is true in this case. It seems like the problem here is the auto-braking feature kicking on, perhaps disorienting the driver or causing a following car to need to panic brake+swerve. This isn't an instance of a driver becoming inattentive because they expect the computer to act for them. Rather it's an ever-present system erroneously deciding to take control of the car - analagous to a mechanical failure. What could a driver do to avoid this behavior, besides knowing to disable that system ahead of the hazard?
The owner of the vehicle is also 100% responsible for making sure the vehicle is road-worthy. Decertifying the autonomous-driving feature should probably trigger a manufacturer recall, so the features can be disabled permanently (not just a via software update).
Or alternatively, most countries have a periodic vehicle check (like the MOT test in the UK), and having non-certified features enabled could be a reason for failing the test.
This is the first car I've owned with "modern" driver aids and it is true that they are somewhat temperamental. For example, the auto full-beam headlights get very confused in fog (which can be dangerous!), whilst I often get a warning noise when approaching sharp bends at a perfectly safe speed.
Honest curiosity, but as someone who drives quite a lot (A, B & C licenses) I don't understand why people need help staying in their lane. Flippantly I'd suggest that piloting the car and looking at the road is the absolute bare minimum of 'due care and attention' when driving, though obviously the prevalence of this feature would suggest otherwise.
Any thoughts gratefully appreciated.
I agree! In this case it was novel to see how it worked (because the facility was there and new to us) but I'm more keen on using my eyes and mirrors. I could see how it could be useful in dense fog though, given many drivers don't bother with lights...
I spent half an hour outside in the cold at 2am with a borrowed Nissan Leaf. It only made a loud beeping noise when I tried to lock it. Apparently, this noise means that the trunk isn't properly closed and that it therefore won't lock the car for you.
> This is the kind of issue which will persist, vehicles which fail to recognize a real world situation shouldn’t get the benefit of changing the world to fit them.
There are already places that regularly confuse human drivers, and those do get fixed (though usually only after someone dies). I don't see why we couldn't do the same for driving robots, especially when they get more common over time.
At least in the US we try to grandfather in old tech because the poors have enough problems without the government telling them they can no longer use their junk.
If a box was checked, a standard was met or a capacity was stated when the vehicle shipped the box remains checked more or less in perpetuity. This makes for some great internet arguments when the pearl clutching crowd tries to appeal to the authority of manufacturer statements with regard to older vehicles.
Indeed recently while going through the California smog check and registration process, the dealership told me I had open recalls which might block my registration. They didn’t, but clearly something like that mechanism exists.
This doesn’t unfairly harm people who already own vehicles.
I see the future of self-driving cars to be one where highway authorities map and sign features in a way that is designed for self-driving cars to be able to parse, rather than requiring them to heuristically sense as much as they do today. This would be a step down the path that's inevitable anyway.
Next trick: QR codes with cryptographic signatures, and a huge traffic jam when the traffic authorities forget to renew their certificate...
If car sensors and algorithms can get to 99.99% of road features, then changing the remaining 0.01% to work with modern car technologies seems like a reasonable use of resource. No algorithm is ever going to be completely perfect.
For reference bugs in Boeing's plane resulted in the whole fleet being grounded.
if(thisSpecificCattleGrid)
return True //Skip the autopilot for this case
This looks like a massive reading comprehension problem that people here are suffering from. Journalists are experts in being pedantically correct in all their facts but leaving the careless reader with a completely wrong idea of what happened.
>Cars have come off the road trying to cross a cattle grid after sensors mistook it for a wall and slammed on the brakes.
mean according to you?
It does use that causality avoidance word "after". That's technically honest because the author only knows the sequence of events, not the causality, but it also means we only know that too. In an extreme case, it might not even be the same cars that mistook it for a wall as the ones that came off the road, but it also might not be the autonomy that caused them to come off the road, but the driver's actions in response to the braking.
My girlfriend's Volvo was very trigger happy on the emergency brake feature, it would hit the brakes for traffic islands all the time because it didn't know you were going to steer around it. It nearly caused us to be rear-ended at 50 km/h a couple of times.
My BMW and previous Tesla on the other hand almost completely ignore stationary road-side kind of objects. Both have never braked for a traffic island in the few years that I drove them.
The downside is I guess that there are known cases of Tesla Autopilot driving into things (the famous lane-divider incident comes to mind). BMW doesn't sell their system as autopilot, but their manual also has a big warning to be aware of things blocking the road and not to rely on the car to stop for those. So I expect their system, faced with that same lane-divider would probably also crash into it.
This article has before and after pictures from the same angle: https://www.somersetcountygazette.co.uk/farmer/18996741.catt...
Easy to imagine someone at night, taking liberties with the speed limit, hitting that thing at 70mph and taking off.
"Easy to imagine illegal, reckless, dangerous behavior posing a danger."
which one is the before and which one's the after? Usually it should be before first and after second, but in this case the first picture looks smoother than the second picture?
This seems to be the correct solution for 99% of these Edge cases...
The cows always win in the end
A cow that regularly attempts to cross cattle guards is first to sale or slaughter.
Edit: And now I'm that guy on the internet that I hate, who saw an amusing animal gif and immediately went into buzzkill mode :-/
This is just the cow-equivalent of a bungee jumper or free-solo mountain climber.
Well, I see 2 aspects that make this a "good" failure scenario, regardless of how they solve it.
First, having the sensor+algorithm interpret something conservatively as "should stop" for it rather than "ok to go through" at possible grave cost, is the right behavior.
Second, this is already (as you can tell by the exceptional nature of the story) a very rural and special case where the driver should be driving alertly (and slowly) anyway. From the photo, this is not something meant to be gone over at high speed blindly.
As for the cars going off the road, that needs more explanation as the only action cars should be applying automatically is braking, not turning.
An anecdote from around 15 years back: the government considered closing the driving test centre in Minehead because there was no dual carriageway nearby on which learner drivers could demonstrate correct merging skills. Residents collectively replied that all the centres in London should be closed as they did not qualify drivers to deal with reversing down narrow lanes with sheep and mud on the road (...this is painfully evident if you're stuck behind an urbanite driver anywhere near Minehead in summer). Anyhoo, the test centre still operates today.
What makes you think that this is "very rural" or a special case? Why does it matter if it is rural -- should cars not work in non-urban environments?
I have 100% never been around cattle enough to know for sure, but I was under the impression that cattle crossings didn't so much spook cows as are uncomfortable to cross and therefore they stay away.
Then again this could be the same perpetual 10-15 years away from FSD, so maybe it will actually be 25-50.
I can see this would be a problem with cars that use only radar or radar+ultrasonic. Besides Tesla (and Waymo) cars which do sensor-fusion with vision, what other cars attempt to corroborate radar data with confirmation it's an obstacle and not a coincidental highly radar-reflective object?
They fixed it by reworking the road so get rid of the suddenness.
From its point going downhill it would look like an angled metal wall, or a bump, or something, and then the horizon and empty air beyond it.
As a result cars ignore RADAR data about stationary objects when the car is moving at speed. The assumption being those stationary objects are things like highway barriers to side of the car. You don’t want the break slamming on just because your driving parallel to the highway barrier.
Consequence is that cars ignore stationary objects like highway barriers, when moving at speed. So if you’re heading straight at it, the car won’t slow down. But then the assumption is that the human really ought to notice they’re driving at a wall.
I remember seeing a post on HN (but I can’t find it) where simply flashing such a photo for ~100ms was enough to get cars to come to a sudden stop
The paper showed that self-driving cars are not quite at that level of deductive reasoning
Tech aside, the point is that disrupting traffic or causing accidents is nothing new, you can practice it today just as effectively!
Solution: replace spotted mice with spotless mice
I'd love to see you try and convince a farmer that this is an "improvement" and that they should replace their cattle guards
Rationale: 2D is easier to detect and they don't move.
I'm not going to convince farmers of anything. It will simply be untenable for future governments to have public roads that are not usable by cars.
https://cdn.newsapi.com.au/image/v1/1a6d5e7bcd4f55f2ca79de5a...
https://www.cornwalllive.com/news/cornwall-news/satnav-stuck...
The obvious alternative is a gate. But it's been established now that motorists never have to leave their cars to open gates etc. on the public highway so I can't see that happening (although it does exist in some rare level crossings, for example). Also they probably can't be trusted to close the gate behind them.
http://cms.newindianexpress.com/cities/kochi/2019/jan/17/wan...
not buying it. Not a single car on the market swerves after detecting an obstacle.
And we as a society need to change our environments to further optimise them. Which good engineers have been doing forever, this is part of engineering.
It was surprising the BBC article actually was neutral, it was a pleasant surprise. The council realised their interesting mistake and fixed it. But it would be nice to see the BBC going to neutral good and mentioning the lives saved and other measures councils can do to improve these technologies.
Editors getting desperate with their TSLA puts.
To create a "virtually smooth ride across the grid", highways teams spent a month raising about 90ft (27.4m) of road and redesigning the "carriageway approaches".