Autonomous Weapons: The False Promise of Civilian Protection (2022)
cigionline.org
cigionline.org
For all of history, military power depended most of all on population size (to provide soldiers) and economic resources (to equip them). Mad wizards were limited by population. However, mad wizards - megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army.
Also, note that these machines are often built for the US military by people who openly denegrate democracy and seem megalomaniacal; they have the technology and a lot of wealth. Could they build a factory somewhere and hide in their bunker until it's over?
I think the problem with these sorts of people is that they are not actually that smart. Cognitively, as all-rounders, they lack crucial faculties of imagination, which not ironically are rooted in compassion, that lead to gross strategic errors.
Landmines and impact grenades tied to a balloon are also "autonomous weapons". They are indiscriminate. The former are persistent and the later are ephemeral. The former are also reusable. Within a weapons taxonomy the most dangerous (for yourself) to manufacture are persistent, autonomous and reusable. In other words, like a knife in a fight, there's a very good likelihood your opponent will take it off you and use it against you. Most IEDs used against British and US soldiers are made from UXO.
Weapons like drones are a gift to terrorists.
For everything you make, one must imagine it being used on a busy city shopping street where your own family are.
It's because the megalomaniacs are not actually that smart that they don't realise taking humans out of the loop is the same as relinquishing control, which is one step removed from handing control to the enemy.
And the inverse also is tautological, of course you can spot those not that competent.
There is also the conversation on what to do about moustache-twisting, cat stroking, comedy Bond villains with tanks of laser-sharks.
I don't think they are fooling any of us. They swagger around in plain sight, naked in their madness, and we could literally write a list of their names here on HN.
Of course these strategic and psychiatric questions are related. "Dr. Strangelove" and "Don't look up" remain useful guides.
The laws of physics do not (or are not known to at least) prevent people from desiring to fix this hilariously suboptimal "democracy" we live under and most everyone seems to hate (well, except when they're gushing about how great it is), but something does.
I often wonder if propaganda and culture play a role.
> I don't think they are fooling any of us.
I believe the opposite - I believe at least 95% of people are adequately fooled (to sustain indefinite disinterest), and most of the remaining 5% are "conspiracy theorists" which we all know can be safely ignored.
There exist people who are both megalomaniac and rational and this is precisely the kinds of people who are typically selected for positions of power. However rare such individuals may be, countries like the US have a large pool of candidates to sift through.
The people in power are capable of being perfectly rational in every aspect, except for the one aspect where they are required to be irrational. The system is essentially selecting people for their blind-spots and that's why we often see displays of extreme hypocrisy from politicians and other public figureheads. They are 100%, perfectly rational about everything except when it relates to pursuing their own self-interest, yet because they are rational in every other aspects, they know how to hide this irrationality from other people.
It includes billionaire leaders of the IT industry. Whatever their flaws, it doesn't stop them from developing, building, and scaling technology - beyond anyone else in the world.
If you have money, buy a sports team, and the managers of the team turn it into a champion, that does not inherently make you a gifted owner or manager - it means you had the capital to get in.
There are other very strong indications of their capabilities. I suppose talent doesn't matter, just their ability to create outcomes.
Attila, Alexander, Hitler, Stalin ... those guys got things done.
"Outcomes" need to be morally qualified.
I think a first step is to re-evaluate "success". The billionaires are not successful humans, as people. They are simply rich.
We have the solutions and have, perhaps not incidentally, discarded them in last 10-20 years:
Postmodernism - yes, I know the reaction to that outrageous idea - is at its heart a distrust of power as corrupting and its application as necessarily evil. Power is the Ring; Gandalf's mission was to stop the Enemy without raw power, but by defeating despair and arousing people's goodness, honor, and courage (e.g., the clearest example being Theoden).
Democracy: No kings, no aristrocrats, no masters; all created equal; liberty and justice for all; every one's opinion as good as another's; everyone is equally trusted to live their own life, make their own choices, choose their own words and religion; everyone gets an equal vote and is a valid juror. To assert greater power was a sin; megalomania was transparently ridiculous. Remember the humility of Washington, Lincoln, etc.
Eary American Protestantism (I'm open to better terms, and I can't define this one well): Humility, simplicity, service to others as the ultimate measure; charity; personal wealth almost an embarassment of greed; the Golden Rule; etc. I remember a New Yorker article about such a culture even on Wall Street in (iirc) the 1980s; high walls around homes in the suburbs were frowned upon as arrogant and not engaged in the community; the head of one bank wore a cheap watch and soon everyone else did too.
We have all that, we know all that. Those who want to seize power try - for obvious reasons - to say they are impossible or to ostracize them. The failure is that leaders who know better - who should remember Washington, Lincoln, Eisenhower, so many more - haven't bothered to pick up the banner from where it was dropped. I think that's all we need.
Prostitute literature. https://www.6parknews.com/newspark/view.php?app=news&act=vie...
Deconstructing the heroic view of history, no matter what the real history is, makes us understand that every so-called strong person can be as despicable as a prostitute, so that we can only focus on what they are currently doing, not on their past identity
The big control in a typical democracy is that the troops of a rogue general will, presumably, disobey them and if the army fractures then the winning side will be the one with more people. Once mechanised armies are in play the numbers and assumptions around that dynamic change.
Democracies don't change human nature, there are still powerful people who want to execute coups in any given society.
There are still far more people in society who don't want a coup, and the desire for a coup isn't binary - democracy changes human behavior, obviously.
The topic was "Civilian control [of, presumably, the military]". The root comment seemed comfortable with the idea that individuals can go rogue if they have access to a large number of machines that unquestioningly obey them. I'm not sure why you think society being organised democratically makes that suddenly a non-issue again, and if you believe that then why did you suggest this topic in the first place?
Military generals are just people and power reliably corrupts. There is a bit of a screen to try and pick emotionally stable and patriotic ones - and I assume that generally works - but the big control is that fact that all the people reporting to them are voters. If the people reporting to them are from a smaller pool of voters and vastly more productive with machine aid, there is room to speculate that control will be less effective or maybe even fail.
Mad engineers could be worse, though.
I don’t know. If there’s a good solution I guess somebody would propose it.
Mercenaries already exist. Commonly.
Though in a developed country it's cheaper to buy lobbyists.
It does not take much imagination to see that the core skill of politicians, persuasion, will be obsolete as well. There will be no persuasion, no speeches, no campaigns. It will be just simply do as they say or perish. It's the end of democracy and the start of a new form of feudalism, but more violent.
If we add brain implants to the mix, the prospect future gets even more grim. If you are important, information will be exfiltrated from your brain without your cooperation.
I've had a novel (as in writing a book) idea rattling around in my brain with something like this.
A distributed AI living throughout the collective internet has humans unwittingly (or select few recruited to wittingly) manufacture a machine army for it.
That being said, autonomous weapons are overall bad, but that explanation is too long for a HN comment.
My understanding of experts I've read over years:
Western militaries, at least, have for over a decade worked on keeping humans 'in the loop', for the obvious moral reasons and also for effectiveness - human-machine teaming is considered much more effective than either humans or machines alone.
Yet they are developing autonomous machines because they don't believe they can rely on communications. The measure-countermeasure contest between communication and jamming leans too much toward the latter and it becomes a serious vulnerability: The machine army could be entirely shut down by simply jamming communications.
We then moved the AI from a simulation to reality and it worked very well. I was an undergraduate at university at the time.
Based on the training data we had (what looked a lot like predictor drone video), I imagine the tech is fully capable and ready to go as needed.
Everyone knew what we were building, but rationalized it as “oh this will be used for speeding tickets or tracking drug deals”… like sure, but I can see the AKs in the video lol
> drug deals
I can't help but think of how the world changes.
Think of speeding tickets, until the 70's, vast portions of the US did not have a speed limit. This changed during the middle east oil embargo with the introduction of the national 55mph speed limit (under the guise of saving energy). When oil supply stabilized and domestic and foreign oil became plentiful, it was not repealed. Now speeding tickets are common. A huge majority of people have gotten them.
I wonder if the tolerance value for going after a speeding car might become smaller as self-driving cards minimize offenses and ticket revenue drops.
In the same way, "drug deals" have led to civil asset forfeiture, causing lots of out-of-town travelers to give up their property, to the benefit of police departments.
so... today's "speeding tickets" and "drug deals" might become tomorrow's problem for a wider cross-section of the population.
They did not have a federal speed limit. State or local speed limits were imposed everywhere. Although pre-1973 limits were often a lot higher than the federal limit.
>When oil supply stabilized and domestic and foreign oil became plentiful, it was not repealed. Now speeding tickets are common. A huge majority of people have gotten them.
The 1973 law was repealed in 1995. And the mandate of setting speed-limits were returned to state-level
>[Well..], both states did have speed limits, but they had sections of rural highways with "reasonable and prudent" speed limits, which were often unposted.
>But they were not without limitations. Drivers were still expected to drive at a safe and appropriate speed for the conditions, and law enforcement could *enforce speed limits based on what they deemed reasonable and prudent.*
So to me than seems even more open to abuse than the current situation.
In a future world where every car is self-driving and obeys the posted speed limit, I can see states and municipalities resorting to trickery and “hacks” like declaring a speed limit that changes randomly and unpredictably, so self-driving cars have trouble adapting and can get cited. It’s not like they are just going to allow that revenue stream to disappear.
Isn’t that just the standard “follow me” feature?
There's tons of work like that (no offense meant) and it never amounts to anything much. If you used Deep RL or imitation learning, or really any of the Big Data, statistical machine learning techniques that are well known, then it wouldn't have worked in practice. Those approaches are extremely unreliable and fail to generalise to new environments robustly enough to be of use in the real world.
The reason why the article above can still ask the question "Have we seen active deployment of killer robots?" is that the question doesn't have a clear answer. If what you did as an undergraduate could really work as well as some people would like, there would be no question left to anyone's mind.
Again, that's not to bash your work. NVIDIA recently announced a new "moonshot" to develop humanoid robots for domestic use. That's because there's no such thing yet. Just a quick glance at the robotics and AI literature should be enough to convince anyone that true artificial autonomy is not yet possible.
most of the team went on (not me) to found a company too: https://reconstructinc.com/
If you're talking about a gun drone that autonomously hunts down people and kills them, then that's not a technological capability currently available, at least not if you're interested in accuracy. Although, I guess that's the point, accuracy is often not a priority. Like early gun powder weapons part of the value of autonomous weapons is their potential for terror.
But is my 1979 calculator artificially intelligent because it can duplicate some cognitive things that I can do? There's a major qualitative difference where we find all the significance in the meaning of AI.
Still, some of the problems with mines, which are illegal or highly discouraged in many situations, are similar to autonomous robotic weapon systems: A fundamental law of war is discrimination between military and civilian targets. Mines are literally indiscriminate, by that legal definition; the worry is that autonomous robotic weapons may be the same. Arguably, they could be more discriminate.
I think one difference is in the 'robotic' aspect of autonomous robotic weapon systems (my terminology): Robots don't need to be mobile, of course, but those with significant mobility seem far riskier and deadlier than mines or artillery. Imagine one that, even from some malfunction, ends up in a city or a refugee camp.
"Anti-personnel landmines are prohibited under the Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on Their Destruction (or Mine Ban Convention), adopted in 1997. More than 150 countries have joined this treaty." - https://disarmament.unoda.org/convarms/landmines/
I’m suspicious that all it will take is one major shock for the house of cards that is international law to come tumbling down just like it did before WW2.
Essentially all of the countries that signed onto these bans either (1) did so in a moment of blinding post-cold war naivety, where they could spin downsizing their armed forces into a good thing for the world, or (2) are too small and/or uninvolved to matter. Anyone else who actually expected to need to fight a war again didn't sign, because they knew exactly how useful these weapons are. The US and SK will never accept a land mine prohibition that includes the minefields of the DMZ, since those minefields are an important speedbump against any sort of ground invasion. Likewise, a cluster munition ban is practically DoA because cluster munitions are the best weapons to attack columns of lightly/unarmored equipment and dispersed aircraft/helicopters. But until February 2022 in Ukraine, a whole bunch of countries (primarily European) had convinced themselves that the possibility of a ground invasion was a thing of the past.