Germany adopts first ethics standards for autonomous driving systems
360.here.com
360.here.com
One thing this would lead to (could be good or bad) is people being “stripped” of their license to practice software engineering. After the 1981 Hyatt Regency disaster in Kansas City, where over 200 people were killed, this is exactly what happened to the engineers involved in building the walkway. They forgot the software equivalent of a line-ending semicolon when they eliminated a single-support bolt for the catwalk accidentally during design review. It collapsed during a holiday party, killing those above and those below. Perhaps as we start seeing more “building computers”, sometimes called “architectural robotics” (a super fascinating subject if you have a chance to look it up, there are already prototypes out there), a software engineer causing a similar amount of destruction might not be so farfetched.
At least until we have a new constitutional amendment getting money out of politics.
Absolutely not. Their original design supported only 60% of the minimum load specified by the Kansas City building code. However, that's not why they changed it. A supplier pointed out that the design was impractical to assemble, and suggested an alternative. The engineers in charge accepted that alternative without doing any calculations about its effect on the load-bearing capacity of the walkways. Had they done so, they would have seen that the new design could support a mere 30% of the minimum load required by the building code. Their engineering process was flawed from start to finish. If I had to make an analogy to software, it would be a company that regularly did development on the production box.
Engineering is not about never making mistakes. Forgetting a semi-colon is common, and any software engineering process that depends on people never forgetting something that trivial is fatally flawed. When you make a mistake in vehicle control software, there should be many layers of failsafes to catch it, such as: the compiler, automated unit tests, automated acceptance tests, code review, manual feature testing (with a simulator), manual release testing (with a simulator), integration testing on a vehicle that has been immobilized, integration testing on a closed track, and production environment trials.
Keep a few mistakes between yourself and disaster. One mistake is a very slim margin.
There are different degrees of software engineering. If you work for a company that makes pacemakers, there are very strict software testing requirements. The same goes for the developers of real time avionics systems for air craft, ECUs for cars and autonomous train systems in places like Singapore or London.
If you're writing software that tracks inventory and ships out shoes, mistakes aren't life threatening and you might have faster processes to get things into production. If you reuse that software for shipping out medical equipment for people with diabetes or sleep apnea, you could run into life treating problems if the testing process is not updated to match the seriousness of the problem you're trying to solve.
That's true, but at some point it's no longer engineering. To practice engineering in my jurisdiction, it's required that you hold a professional license and work for a company with a permit to practice.
I am a professional software engineer (P.Eng.), but many employers for non-critical software do not have a permit to practice. They have no need for it.
I need to file my Continuing Professional Development hours every year. One category of hours is just "practicing engineering". However, if I am working for a company that does not have a permit to practice, it would be quite a problem if I claimed I was practicing engineering.
Practically speaking, that means software engineering is just the development of software which could be dangerous (at least where I live). It's distinguished from general software development mostly by the need for rigour.
I should also note that anyone can assist an engineer, so it's not like Computer Scientists are cut out of the field. Many disciplines have technicians and other specialists who are just as technically capable as the engineers in their domain (often more so).
The lead engineer signs off of on releases, and they're the main person held responsible if there's a safety incident traced back to sloppy work. That was the only real difference between the engineers and non-engineers I worked with. It basically just meant, "the buck stops here."
> Keep a few mistakes between yourself and disaster. One mistake is a very slim margin.
This is all spot-on. In a related vein, I used to despise any mention of "Systems Engineering", and then I worked in the development of complex systems. "Systems Engineering" is really about mindset, tools, and processes to impose coordinated rigor in the engineering of a given system. There is no Theory of Systems Engineering in the same way there are theoretical underpinnings in Mechanical, Civil, Electrical, Aerospace Engineering, etc. Elevating it academically to a fully-fledged field of engineering in the same way as these is the irritating part. However, the likelihood is extremely high that with any large and complex physical system that embodies a strong degree of correctness, good systems engineering practice was integral to its development.
I do not think the majority of software development today corresponds to real engineering practice for any rigorous definition of "engineering". In general, it is much closer to the practice of a craft, like fine carpentry or auto body work (c.f., Eric Sink). What you describe, very aptly, with multiple "mistakes between yourself and disaster" and an embrace of rigor, verification, and deliberate effort undertaken to compensate for the lack of inherent rigor in things like our programming languages, is engineering. Writing lots of code does not make one a "software engineer" any more than building a table in my wood shop makes me a mechanical or civil engineer. If you cannot explain how you develop systems where the implementation in code is just the final manifestation of a design, and that you specifically have systems in place to compensate for or screen out implementation flaws (especially the ones your implementation tools cannot eliminate by construction, e.g., many of today's commonplace programming languages) and logical flaws, then you are not practicing software engineering.
Our ability to construct large and complicated systems in software today far outstrips our ability to rigorously guarantee that the software satisfies correctness properties. This is fundamentally unlike what systems engineering with traditional physical engineering disciplines can provide. I think what Germany is doing is here is right, and fundamentally different than all the stuff going on in the U.S. with Tesla, Uber, Waymo, etc., that seem to be doing huge amounts of ad-hoc implementation, with no consensus on high-level properties the implementations should verifiably satisfy. The raging debates in the U.S. seem to be about whether or not states should allow systems to be tested on public roads rather than what the provable properties of these systems should be.
Nitpick: no.
> [Dr.] Margaret Hamilton promoted the term "software engineering" during her work on the Apollo program. The term "engineering" was used to acknowledge that the work should be taken just as seriously as other contributions toward the advancement of technology. Hamilton details her use of the term:
>> When I first came up with the term, no one had heard of it before, at least in our world. It was an ongoing joke for a long time. They liked to kid me about my radical ideas. It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware. Not because of his acceptance of the new "term" per se, but because we had earned his and the acceptance of the others in the room as being in an engineering field in its own right.
https://en.wikipedia.org/wiki/Software_engineer#Origin_of_th...
Emphasis added.
That’s not going to end well. As we saw in the AZ crash, when there’s a situation at hand, it’s hard to transition the attention properly quickly over to a human who has been not paying attention because while the car was handling everything just fine. It’s unrealistic to think that’s possible in anything less than at least 10 seconds.
More interesting is the implication that this requires the possibility for humans to regain control. So we should not build cars without control mechanisms (steering wheel, brake pedals, ...)?
When the driving is easy, the AI _can_ do it. We are by default discussing the unusual, difficult situations.
In a fight between people writing down how they want things to be, and basic human psychology, the latter has won every single time.
"In emergency situations, the vehicle must autonomously, i.e. without human assistance, enter into a “safe condition”."
What they mean is that the driver should have the freedom to overrule the system, not that they must do so.
That's what I call an intense first-time-setup session.
As this was a technical fault you suffer no legal repercussions, but carry the mental despair over whether you answered the setup question correctly to an early grave.
Fast forward couple months, and your morality profile is yet another thing advertisers know about you.
It is however worth trying to encode what happens in emergency situations to be as similar as possible to the local highway code, so you've got a sound basis for decisions. And it's very useful if the vehicle can "explain" its actions afterwards.
I recall a study (and would be grateful if anybody has a link, as I cannot find it at the moment), which tested the reaction times and decision-making of drivers who had just been handed the wheel by an autopilot, in a simulated driving situation. What they found was that it took about 10 seconds before the drivers got up to the level even of a drunk driver.
To me, that study indicated a fatal flaw in the entire concept of hanging control back to the human driver. If there is an emergency, it will be resolved in 10 seconds one way or the other. At that point, you can either have the confused computer trying to drive, or the even more confused human trying to drive.
https://www.zeit.de/mobilitaet/2017-02/autonomes-fahren-auto...
For example, the car is driving me somewhere, and there is a wreck on the other side of the road. Foolish human nature takes over, I take control of the car to slow down and look at the accident. I am overriding the cars autonomous software. The car in front of me is doing the same thing, but slows down even more. My car senses a collision about to happen, and takes control by applying the brakes, ignoring my input on the gas. Once the clear danger is over, the car must give control back to me, but only if I continue to choose to override its normal operation. Otherwise it will resume driving itself.
This is basically an application of Asimov's "three laws" concept.
I'm not saying that to be reasonably safe and/or viable these cars need to be piloted at a level akin to say a Mozart on the violin or a Peart on the drums, however that level of Genius is achievable. So again I ask: do you think the brain trust behind the R&D has any inkling of precision driving on the edge of adhesion? After all, it does rain and snow out on the Boulevard.
BTW this is what a virtuoso can do in the most difficult car there is to drive, a Grand Prix car. And yeah, I loved the guy... so I picked a vid with some fluff.
So from the perspective of a startup trying to get in this market, this sort of environment might prove superior, because your business risk is much reduced. You know what sort of standards you must follow and as long as you follow those standards you do not have to worry about being blamed for negligent murder by the family of every accident victim.
I like it how it shows that data your device generates is your data, and only yours. It's like stating the obvious.
And, you'll likely have to sign a waiver granting the manufacturer access to your data or they won't sell you the car.
[0] https://en.wikipedia.org/wiki/Informational_self-determinati...
The answer really depends on how you see cars interacting with the rest of society. To what extent should we compromise the safety and rights of other people to try and enable fast and efficient car journeys? That is highly contentious even before you introduce self driving cars. These kind of guidelines don't answer that question.
Maybe then with a built in logging system that says every time control changes, and deletes the record when irrelevant (like three changes back). Then just take the data whenever there's a car crash, and hey Bob's your uncle.
So in most cases, the likes of Tesla or Uber or whoever pay the other party's insurance. If it can be proven that a certain programmer or team screwed up, then they're personally held liable and sent to prison if necessary.
That'd certain encourage decent standards here. Do your job poorly and have it tied to your actions, and you go down for 20 years.
But put developers in prison for 20 years because the car crashed, that's untenable. You wouldn't have decent standards, you just wouldn't have anything at all. What kind of developer would put themselves in that position? Everyone makes mistakes. All complicated software has bugs. You would only have software made by fools, and marketed by fools, that was much more dangerous.
If someone makes software that reduces accidents from 10% of drivers per year to 0.1%, they still cause 100,000 accidents a year in the USA, maybe a few thousand fatalities. Give them the electric chair right?
If you are going to punish them for the mistakes, will you reward them commensurately for success? For every life they save, can you give them a life?
I'm curious how this blog post by HERE (9 months later) differs from the earlier official statements.
I was watching car accident videos on youtube yesterday (I got sucked in and couldn't look away. I lost two hours.)
Almost all of them were due to someone doing something obviously stupid or wrong: turning without checking for oncoming traffic; passing unsafely; over-correcting and then losing control; mis-judging clearance with stationary obstacles; tailgating; etc...
It occurred to me, what if we program the robot cars (and can we please call them "auto-autos"!?) just so that they won't engage in dangerous behaviour? The basic idea is, if the way is not clear, slow down.
What I mean is, it may just be Too Hard to make a robot drive a car like a car; but it should be already possible to make a safety vehicle that prevents human error.
This makes a certain kind of sense if viewed totally in a vacuum, apart from any other considerations. But life is not all about the car. Is human choice unimportant? What about freedom to travel where I want and when I want? Is curtailment of those and other freedoms to be sacrificed based on statistics? Whose statistics? How will these statistics be gathered? Who will interpret them? Through what lens?
This sounds like the kind of reasoning that made it illegal in many jurisdictions to let your kids play outside on their own.
Expensive is a feature in this case. It ensures that stuff gets tested well before companies even consider having it double-tested by the govt.
As you're apparently aware, minister Dobrindt (who was in charge of both) took great pains to deflect trouble from the German automotive industry.
I think history will show that by making their present more pleasant, he'll have made their future less stable. Now they get to milk the status quo for a while longer, instead of being forced to adapt to changing times.
For this and other reasons, I despise the man.
* wikipedia claims ongoing investigations against 37 people involved
* Politicians are working on the Musterfeststellungsklage, a class action equivalent for the German justice system. Explicitly so that buyers have better odds suing VW.
The trolley problem as bystander is a different thing and generally not acting is the legally safer option though in both cases the likely sentence would be minor through various means.
edit/additional note: while you can't shoot down the plane legally and no soldier will be obligated to follow such an order, they would likely also fall under the "Übergesetzlicher Notstand". It's not a legal basis but simply a defense.
One guideline states that an attempt at reduction in the total amount of human damage is defensible (Rule 9, pg. 11), which supports killing 5 people.
Another (Rule 8, pg. 11) states that genuine ethical dilemmas, such as life-vs-life situations, should be judged later on a case-by-case basis by a court because there are no concrete moral guidelines by which to ex-ante program a computer by, referencing that a human driver could pick either choice in such a dilemma and perhaps act illegally but not necessarily culpably. I take this to mean that deviating to kill one person could also be reasonably defended.
How does this make sense? 5>1 therefore _diverting_ to hit 1 person instead of 5 is reducing total amount of human damage.
There is no trolley problem. When vehicles are on the road and operating according to the rules of the road, there is no choice about whether to swerve to avoid something at the expense of another. Cars are not being commanded to leave the road to avoid crashing into something on the road. You stay on the road, stay in your lane, and don't get involved with people on the sidewalk, etc.
If there's a bicyclist, motorcyclist, etc, they are just as much a vehicle as you are, occupying their position in another lane, or in front/behind you. You stop in your lane, change lanes if safe, and that's it.
There's no, "sacrifice 1 to save 3" kind of argument.
But I do agree that I don't see such scenarios ever occurring in reality because computer controlled cars are not going to get themselves anywhere near a trolley situation. They'll put the brakes on long before a human would ever recognise the situation as catastrophic. Almost all unexpected emergency situations are straightforward: the correct response will be to brake hard; the consequences are not ethical but rather dictated by physics.
Swerving onto the footpath is an absurd option for countless reasons and cars shouldn't be allowed even if the algorithm is fairly sure the footpath is clear.
The closest plausible scenario to the trolley problem I can imagine actually occurring: "An unidentified object has fallen onto the road meters in front of me, and I am boxed in with cars to my immediate left right and rear so I can't swerve. Do I break hard and probably get rear ended or do I break soft and definitely crash into the unidentified object?"
My point was more that your responsibility should be to avoid crashing into things. In the posted scenario, you're probably better off slowing down and being rear-ended than smashing into whatever and then _also_ being rear-ended.
On a slightly different point, if someone's following me so closely that I'm properly worried about their stopping distance, I will slow down and let them pass. Better that the idiot is in front where I can see them properly and they can't crash into me, and then I don't have to worry about the car behind me if I need to brake suddenly.
Every time I see these tiresome discussions about "should we save a child and kill a pensioner", it's a signal that the speaker has no idea how autonomous vehicles work.