On Robots Killing People
schneier.com
schneier.com
This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck.
Schneier falls into this same error, saying Tesla Autopilot sometimes "ignor[es] red lights, stop signs, and crosswalks." Tesla isn't perfect, but I'm pretty confident there's no code in Autopilot that says, "yeah, I see that crosswalk, but Elon's in a hurry, so let's goooo!" -- okay, maybe in Elon mode. But not in general. So no, Autopilot doesn't "ignore" those items: it misses them.
Assigning intention to robots at this point is pretty much equivalent to assigning intention to a clock because the hands go around. We have a long way to go before we can/should assign intentionality to "robots" this way. Of course, at the rate things are going, "a long way" could happen in the next few years. But it's not happening now. I'll happily consider counter-examples.
The essay suggests "Any regulation of industrial robots stems from existing industrial regulation, which has been evolving for many decades" & that regulations need to be clear about other robots as well. Interestingly enough, if we applied existing regulations to Tesla's Autopilot as we would to a driver who racked up as many fatalities and crashes (736 crashes and 17 deaths in the U.S. since 2019 [0]), they'd be behind bars long ago, let alone never able to drive again.
0 - https://www.caranddriver.com/news/a44185487/report-tesla-aut...
Aat this stage I agree that we are dealing with "blind machines", with no more "intention" than a steam engine.
However, the central argument in Schneier's piece seems to be that robots inherit the intentionality of their creators.
If I create a machine that kills someone, my "intentionality" is fair game to be dissected in court, and ultimately I may be found guilty as if I had killed with my own hands.
That is the kind of high bar I think society expects, and is certainly one that should scare the crap out of robotics developers (aside: I have been one) to ensure the highest standards of engineering discipline and industrial ethics.
It's definitely fair to consider the 'intentionality' of the idiot too.
It's still on leadership if they choose to hire idiots who get hurt. The solution is: don't hire idiots.
If you look historically at manufacturing and the origins of signage and safety training, you'll find them mandated by the government (i.e. OSHA, etc.) because those things weren't always voluntarily provided by the employer. Upton Sinclair opened our eyes to this 100 years ago.
When you run a shop with five experienced people, if each of them has a 1-in-a-million chance of getting seriously injured on any given day, each year you have a 1-in-548 chance of a serious injury. Any sane person would call this risk acceptable.
If you run a shop with 5000 people of equal skill level to those five above, you're talking a couple of deaths or permanent disability injuries per year, just from the numbers. Any rational person would say hey, that's the same risk, why are you mad? But society isn't rational and if any legal entity no matter how large kills or maims a couple of workers a day, there's an outcry and executive heads will roll. So yeah, they try to do better. At this point maybe 'hire smarter people' works?
Now imagine you're running a multinational company with 500,000 people. If you operate like that original shop you'll see a couple of hundred deaths in your first year and then get sued into oblivion or lynched, and remember those were skilled experts. There aren't 500,000 skilled experts available for hire, and most of these jobs are ahem less skilled anyway, so you're hiring people with much higher rates of personal injury. To operate at this level you need to do stuff that seems dumb to the uninitiated, like relying on layers of subcontracting companies for everything while implementing byzantine and ever-shifting safety regulations that each subcontractor must commit to following. Sure, it costs an order of magnitude more, but it's worth it for the severability.
And the accidents that I know of, are where the security mechanism have been tampered with. A proper designed and certified german machine is hard to get injured by if followed protocol (and the reasons are past accidents and suing the manufactor).
The weapons industry (and its lobby organization, the NRA) would like to have a word with you...
So not all that weak, really.
Sure (and thanks for pointing this out) but this isn’t a philosophical piece, it was written for the Atlantic, and he’s being careless with his language.
The second one seems fair: he is describing the result of what the Tesla Autopilot is doing. I do not think that he means to say that the Autopilot is understanding that there is a red light but then it makes a consciuous decision to itgnore it. He is just saying that the robot is malfunctioning, as it should be clear from context:
"Today’s AI-related robot deaths are no different from the robot accidents of the past. Those industrial robots malfunctioned, and human operators trying to assist were killed in unexpected ways. [..] Malfunctioning Teslas on Autopilot have deviated from their advertised capabilities by misreading road markings, suddenly veering into other cars or trees, crashing into well-marked service vehicles, or ignoring red lights, stop signs, and crosswalks".
Even more likely, it’s a manufacturing robot, with no vision at all except maybe for the pieces it is manipulating. It almost certainly has a line on the floor, or a railing even, to keep people out of its way.
In short, it has zero obstacle avoidance. I’d be interested in counterexamples.
> The second one seems fair: he is describing the result of what the Tesla Autopilot is doing. I do not think that he means to say that the Autopilot is understanding that there is a red light but then it makes a consciuous decision to itgnore it. He is just saying that the robot is malfunctioning, as it should be clear from context: "Today’s AI-related robot deaths are no different from the robot accidents of the past. Those industrial robots malfunctioned, and human operators trying to assist were killed in unexpected ways. [..] Malfunctioning Teslas on Autopilot have deviated from their advertised capabilities by misreading road markings, suddenly veering into other cars or trees, crashing into well-marked service vehicles, or ignoring red lights, stop signs, and crosswalks".
He describes Teslas as “no different” from non-intentional machines of the past, and then describes them with “intentional” language.
>I do not think there is a more neutral way of describing the events.
How about, “Industrial accidents happen through lack of human foresight and carelessness. The same is currently true of ‘AI’ products like Autopilot. But the time will come, and probably soon, where AI alignment and intentionality will be part of the conversation. We should prepare for that starting now.”
Not as catchy as “Robot rampage!” But more accurate and, as you say, neutral.
I really really don’t think this is an article about convincing robots to be nice to us (AKA alignment); I think it’s an article about how we need stricter regulations for software that controls physical machines. Doesn’t that sound like a good idea?
I’m not arguing against the underlying point, I’m just disagreeing with how it’s presented.
Reasoning is actually pretty good now. An AI can compose a sonnet on electron degeneracy pressure no-problem but ask it where the red ball is - and really inspect its localization of that red ball and it’s way off. Orders of magnitude of imprecision.
A modern AI can reason through the trolley problem and millions of variants. But it can’t see the person on the track with very high confidence, or be sure that the trolley is going to hit the person or not.
AI systems are quite good at perception in a traditional sense (e.g. determining a red ball is in the sensor receptive field), but once actual cognition about how the red ball relates to the world around it becomes important, current approaches are limited.
I would argue that the first wave of perception is solved much more completely than the ability to reason about complex interactions of objects grounded in the physical world.
Disclaimer: I am a robotics research scientist
A red ball detector that operates at 60hz with a 99.9% accuracy will make an error on average once every 11.5 seconds. I think the issue is not so much the interactions of objects and the world but rather we're just wildly off-base regarding the base performance needed.
Ignoring can mean failing to notice something, as well as noticing but disregarding it. Words can often have multiple similar complementary meanings. I don't think the usage here is incorrect, or even misleading.
You don’t think it’s important to draw a distinction between whether a robot was deliberately programmed to do something malicious as opposed to a bug which caused an oversight? Come on.
https://dictionary.cambridge.org/us/dictionary/english/ignor...
> to intentionally not listen or give attention to:
> to give no attention to something or someone:
Two definitions - one with intention, one without
> words mean things.
Yes, but they often mean multiple things. Very few dictionary entries only have a single definition for a word. You can't use "words mean things" to claim that "this word means only what I say it means, regardless of the other accepted uses it already has".
> You don’t think it’s important to draw a distinction between whether a robot was deliberately programmed to do something malicious as opposed to a bug which caused an oversight?
I do think there are times when it is very important to do that.
I just don't think that "ignore" is the right word to make that distinction, and isn't a distinction the authors were trying to make at that particular time, because if they were they would probably have used another word.
...at least, that's how I read it. It seemed clear to me. ¯\_(ツ)_/¯
https://dictionary.cambridge.org/us/dictionary/english/ignor...
> Definition: to give no attention to something or someone:
> Example 1: They ignored our warnings.
> Example 2: The mayor ignored the hecklers and went on with her speech.
> This seems inflammatory and highly misleading. I could be wrong, but I highly doubt it: the robot didn't "determine" anything. It had assembly instructions to follow, and it followed them. The worker got in the way and was struck.
I was prepared to argue with you, but upon reading the article, I agree that the description you quote makes a huge category error.
I think it's believable that we'll soon get AIs that are advanced enough they can do agent modeling and that some subgoal-maximizer accident could happen where the robots determines "the most efficient way to accomplish <objective> is to take out that human so he doesn't turn me off", in which case the article's poetic description would be correct.
Except the quoted paragraph is talking about an industrial accident in 1981, so the whole "eliminate the threat" presentation is obviously absurd.
To take just one example, software makers are uniquely able to shield themselves from liability via EULA. I can sell you software for millions of dollars, it can say “reliable, helps route 911 calls” on the tin, but it I make you click an EULA as everyone does you can’t sue me even for the most basic negligent defect in the software.
Indeed, we tend not to even use words like “defect” when discussing software. We use words with more forgiving associations like “bug.”
We do not have any widely used metric of software quality. Even hypothesizing such a measure feels a little absurd in todays climate.
It’s basically still the Wild West in software. Even as the systems controlled by software become more critical and the those critical systems become more widely deployed.
Internet connected vehicles plus control of steering, acceleration and braking via the vessels computer is already an existing problem. Some researchers got into trouble some time ago for taking over remote control of a vehicle on a live highway. There are also videos of people mysteriously accelerating and driving into buildings whilst the driver is trying and failing to stop the vehicle. I'm not linking the ring videos of Anne Heche but those can not be unseen.
Before we can secure robots and prevent them from taking people out I think we first have to secure their predecessors of which they are based on. If a trusted partner or random skiddie can execute malicious undocumented codes or add their own malicious code then the game was over before it started in my cranky opinion. Cars and their remote networks get released and patched just like existing internet services and have a long lag time in between patches giving malevolent parties plenty of time to erase drivers. Also missing from these automobiles is a truly-tamper-and-fire-proof "little black box" that records every instruction executed by the car for the last 72 hours, including the undocumented or hidden instructions.
And you?
As for me, I waited until I was sure I wouldn't kill myself to buy a motorcycle, which for me meant "last year, at age 52." Sure, I still gotta worry about everyone else, but that's a risk I've been taking on a bicycle for years already.
Mine is a Royal-Enfield Classic 350, which is perfect for 85% of what I'd want to do -- errands in town, etc -- but is obviously limited to surface streets and slower back roads. It tops out at 70, notionally, but it's not super happy above 55(mph).
We'd dropped to one car during the pandemic, because my wife has traditionally taken the bus to work. I'd have the car during the day for the odd daytime errand. Then she got a better job that required driving to work, and we really really didn't want to buy another car - and I had always wanted a motorcycle... She wasn't super excited, but the price tag helped a lot ($4800US).
It's lovely, and I get comments on it constantly, but after a year I'm definitely starting to think about Next Bike. Top contender is a Bonneville T120.
See: self-driving cars.
The more challenging topic is military/government use of robots in war or citizen control scenarios. Areas like that seem very relevant if you look forward a few decades given the pace of progress of Boston Robotics and AI.
See: Terminator, the movie. Time travel aside... :P
Absolutely not challenging. They will come and will be used. Nothing you can do about it unless you DRM everything. And then of course the government will still have them. Topic closed.
>Interesting that the article focuses on the unintentional killing of people by robots
I doubt that there was ever killing. The first two examples are people trying to do maintenance on plugged machines. If you try to fix moving steamroller by standing in front of the wheel - don't be surprised at the consequences.
The south africa one - it depends - if the turret just misfired in this general direction - is no different than a mine or shell exploding. If it scanned the soldiers, recognized them as targets and then shoot - then it is killing.
But already programmers have intentionally written code to react when facing this dilemma. We just don't know what code they put in there.
Example: a truck stops in front of the autodriving car, too quickly for it to stop in time. It knows this; it understands its vehicle handling to many decimal places.
So what should it do? Continue futilely braking to a likely-fatal accident when it impacts the truck? Or swerve to avoid?
Swerving may involves other people now: people standing by the side of the road, or emergency vehicles and personnel at work. Even the people in cars behind, auto-driving or not they are part of this. We hit the truck; they hit us. Or we swerve; they hit the truck.
You see? Kill my passenger, or potentially more by saving my passenger.
It's not simple, and not obvious (thus the longevity of the philosophical puzzle). But now robots are making the decision.
We can make laws to force the programmers' hands. Then it's no longer the intentionality of the programmer we have to examine. It's the rules and regulations we put in place, the intentionality of the regulator, of the standards committee. Ultimately of voters.
My humble opinion is, autodriving cars should never leave the roadway. See, the people on the road have voluntarily put themselves 'in the game'. Kids playing by the side of the road are not part of this; they should under no circumstances ever be involved if possible.
Anyway, the trolley car problem. It sucks. It's here.
That is physically extremely rare - trucks have lower deceleration and higher breaking distance than high performance passenger cars. Also why TF is not the autodriving car keeping distance allowing it to break in time. The focus should be on not getting into shitty situations - and because AI attention is unlimited, we can do all kinds of sensors and also mesh the cards, couldn't be bored and is vigilant 24/7 it should be theoretically possible.
Just speculating, but I'd wager we end up with a suboptimal solution to particular instances like this in favor of a general policy that is a) predictable, and b) minimizes liability. IOW, the car will be programmed to brake rather than swerve, because swerving opens many unpredictable cans of worms, as you've noted. Braking may be the worse answer here, but as a general matter, I think you're going to have an easier time in court arguing that the car did its best to brake to avoid a collision vs. arguing that the fact that it swerved and, say, hit a schoolbus or went over a cliff.
In practice, these days most likely a neural net decides what is the best maneuver to avoid serious harm to the people in its car. As alternative the net could try to enforce rules that "global harm" should be minimized.
And then both options as in the trolley problem are most likely not equal, in terms of probability.
I think this is as close to the "correct" answer for the Trolleycar problem as one can get. Even when your passenger is the Nobel winning cancer scientist and the hapless bystanders are "worthless criminals" (or whatever arbitrary utility standard we suppose society might choose).
Restricting the fallout to those who choose to use autonomous devices while totally protecting "externalities" also aligns with market forces, if you believe in those.
The autonomous car absolutely should kill the passengers, even if a better utilitarian outcome seems available but involving others who did not choose to play that game.
And herein lies the current deep fault in all technologies (but particularly digital ones): The convenience of the few (or even the majority) is becoming an imposition on the welfare of others who do not choose or want to play that game. It is, in these terms, quite simply "tyranny".
Not sure if it matters for your example, but if that happens to a human, it's his fault for not keeping a safe following distance. It could also be the truck's driver fault if the sudden stop is not justified, but in the case of the robot, it should be programmed to keep the distance.
After all, once we get rid of those pesky humans, shouldn't cars be able to barrel down the highway at 100mph with the following distance of stop-and-go traffic, because of the magic of internet and algorithms?
Regardless of fault there's the issue of programming are response, which was the point here.
I used to drive to work every day, very busy highway, and faced an annoying dilemma. If I kept a safe distance, the driver immediately behind me would pass me inserting himself in the gap, forcing me to brake and leaving more distance, inviting a new vehicle to pass me. Or I could tailgate and risk being part of the next multiple colision accident. After seen a couple of those, no thanks.
This is the kind of human idiocy that robots were suppossed to avoid. But if they're going to mimic it, what's the point? We will be competing with them and sooner than later, we will see how big companies behind them will outmanouver us, also in courts. So not as optimistic as you about regulation.
So we can only trust in probability and do the best we can. Robot or human, same issue.
Some off-road hazard can inject itself onto the road without notice. All vehicles would be creeping along the freeway if they wanted perfect safety from such events.
That's a tricky reasoning line. Safe distance is not intended for those cases. The point is that, given a colision, it won't be propagated.
About congestion, it doesn't seem a problem if you can drive fast enough to make an issue of keeping the distance. Bottlenecks come mostly from exits.
The roads are currently full of human drivers, and ~zero of their driver-training programs include anything resembling trolley problems.
A reasonable programming goal and standard for autonomous robots would be: "Behave like a human of decently-above-average skill, training, and safety-consciousness".
And, this argument doesn't address at all what to do about this well-understood traffic situation and the programming we will accept.
...any case where the lawyer knows there could be Deep Pockets in play. Quite true. And there is a wide gulf between "X would be a reasonable engineering and regulatory standard", and "X is the current standard".
OTOH...
> Example: a truck stops in front of the autodriving car, too quickly for it to stop in time. It knows this; it understands its vehicle handling to many decimal places ... a likely-fatal accident ...
This is not a "well-understood traffic situation", and it may not be physically possible without a crazy assumption or few. The autodriving car has brakes, a crumple zone in front, air bags, etc. Cars generally have much shorter braking distances than trucks, and the autodriving car presumably has quick reactions. Was the autodriving car blindly approaching the truck at already-lethal (even if the truck had not stopped) speeds? Is the truck loaded with high-sensitivity explosives, that a fairly minor collision will set off?
Citation needed.
> was derided on this venue for unreasonably introducing the 'trolley car' philosophical
Appropriately.
It would be more helpful to post comments that illuminate the issue or introduce new ideas. "You so stupid" is not one of those posts. Try to do better.
The unfounded assumption that I was challenging was the existence of trolley problem solvers being deliberately written. Nobody is doing that.
Are people (and kids) crossing the street at a marking "in the game"?
Military organizations globally are now working at breakneck speed to create *literal* “terminators.”
I know cause I make them.
Replicator is skynet v2…we tried it and weren’t ready with MDC2 in the teens ABMS etc are the follow on attempts.
Every “unmanned” “cooperative” etc… system are the terminators
As Bertrand Russell said : “until there is no more war, all technologies will be used for war”
But, beyond mashing people's emotional buttons, how exactly do realistic terminator robots differ, morally, from many of the robot-free (but lethal) military technologies which were widely used a century-ish ago? - mass bombing of heavily populated areas, artillery barrages, minefields, etc.? Germany alone produced almost 2 million of their S-mines for WWII.
Don’t worry, the pentagon is totally incapable of breakneck speed.
Alas, it's about the unintended dangers of ever sophisticated and fully automated tool use. Is this satire?
Yeah, something heavy on wheels can roll you over, crush you and kill you.
I'm not trying to dismiss any safety regulation but isn't the big elephant in the room one of the main driver of all technology: military use?
Maybe most people gave up on that and try to focus on things in the civilian sphere that they can potentially impact. A "healthy" compartmentalization of sorts.
But for me the contrast is so stark that the examples mentioned in the article are unintentionally comical and forced, even if the underlying questions valid and transferable to military use.
That seems awfully late?! Germany already had the "Society for the Supervision and Insurance of Steam Boilers" since 1866, and according to https://en.wikipedia.org/wiki/Technischer_%C3%9Cberwachungsv..., "from 1871, membership in such an association exempted them from inspection by a state inspector" - so at that point, there was already a system of state inspection. And all this emulated a system that was already established in the UK, so the concept is even older. But maybe it's the American attitude to regulation that delayed similar regulations for so long in the US?
> The examples are more about Machines killing people than Robots killing people... The watershed will be when robots are build to kill people and then actually do kill people the intended way, but not necessarily the intended targets.
I presume the GP is talking about the case when the robot just goes "Nah, I'm going to ignore your target and kill these other people over here".
Serves 'em right for not being able to provide value to capital. /s
https://thebulletin.org/2021/05/was-a-flying-killer-robot-us...
As far as the effort to regulate LLMs, it seems more about the desire to keep them from generating 'bad ideas' or teaching people how to do bad things (which they can already learn how to do in any library or on the internet already, incidentally). Unaligned LLMs are not likely to go around killing people, although they could be very offensive.
Wider recognition of the fact would do good to the quality of the discourse. What exactly are we scared of, considering that we had decades of experience with these systems? How much truth is there to those fears? How much of it is shallow media driven emotion vs genuine risk?
Also sounds like he's blaming the tool, instead of the operator of the tool. I really expected that number to be higher when you account for:
Have all the operators had proper training to even work on this machinery?
We'lre all the safety features built into the machine from the OEM being utilized? Alot of the additional safety features are disabled to make it easier to run. Also no training is needed to have someone press the run button.
We're there auxillary safeties installed? (Light curtains, safety scanners with properly set warning and red zones).
A lot of times machine operators have no control over the work environment or equipment integration and are told to just get the job done with what they have.
I do not believe we need one.
Farm implements have been killing us for far longer, and in far larger numbers. (approx. 20 deaths per 100K, out of 2.1mm in agro in 2020. [1]) Yet, we do not have a "tractor overturn death prevention agency (TODPA)" There is already ROPS 1928.51.
In the USA, re-evaluating the OSHA requirements would reduce the number of at-work robot accidents.
[1] https://www.cdc.gov/niosh/topics/aginjury/default.html caveat emptor lots of lumped in deaths
How about "Safety Alliance for Farm Equipment Turnover Risk Elimination and Awareness Division (SAFE-TREAD)"?
:D
You can say that again. Also the first part ("technologies to self-servingly shift the burdens ... onto society at large") is frequently the very innovation they're talking about in the last part.
It's a welling-up from mysterious depths and only the closer tip is "rational", if at all.
And no, the machine does not access these depths (which is kind of the whole point of machines) and has no "intentions".
For example: the Chinese have created a massive surveillance network of CCTVs and drones feeding data into the AI-augmented social credit score system. These are the AI technologies of theirs that we know about. I wouldn’t be surprised if they have lethal drones controlled by AI as a military project. We’ve seen time and time again that the Chinese Communist Party are perfectly happy subverting anything and everything in the service of upholding their own power. It’s a no-brainer to assume that AI ethics are just one more line item in the laundry list of the things they’d subvert.
I wonder about the world’s capabilities to counteract such a ruthlessly amoral threat without a deep understanding of the technologies they use, and how to counteract them.
That sentence right there makes me angry. Not at Schneier specifically, but at this blatant lie that somehow automation is responsible for even a fraction of job loss, unemployment, and the deaths that ensued. Wake the fuck up, people: it’s not robots, it’s capitalism.
I was born in 1982. Many people my age still remember being promised that machines would lift the burden of work from us, that we’d all enjoy 3-day work weeks and so on. A huge amount of automation did happen, but the rest, not so much. There are many reasons for this, but a big one is that fewer and fewer people concentrated a bigger and bigger share of the profits. Now when we automate something, it’s not, as we were promised, to reduce the physical or mental burden of workers. It’s to reduce the costs associated with wages.
The only way Schneier’s sentence makes sense is if we can’t change our economic system, but for some reason we can stop or limit automation. But even that isn’t going to work: capital owners are going to continue automating stuff as long as doing so increases their profits.
(Edit: I guess that kind of criticism is not exactly welcome in a startup forum.)