Hacker News Reader

Comment by roenxi | Hacker News Reader

A bit more scary is the political consequences. One of the strongest controls on rogue generals is that a fair chunk of their troops will mutiny if told to destroy their home country for said general's benefit, and logistical support would vanish.

Armies of killer robots start radically changing the calculations. It isn't obvious how but it probably removes a lot of the interest of more ordinary people and makes it a question of what the elites want. If the programmers are on side and the factories are mostly automatic, then the ability of people to push back on their own military becomes complicated.

View on HN

>> that a fair chunk of their troops will mutiny if told to destroy their home country

I'd be careful about this. There is a general principle in military communities that troops are not to obey "illegal orders" but time and time again throughout history we have seen that backstop fail. At a base level we all do what we think is best for ourselves. Troops will shoot prisoners. Troops will shoot protestors. They will nuke their own hometowns if they believe that is either necessary or if failure to do so will result in their own demise. And many troops, people, will do what they are told out of pure loyalty to a charismatic leader. The only real safeguard is layer upon layer of checks to ensure no one leader can ever exercise independent control.

It's a fair point that humans don't magically make this go away, but the thrust of the argument above is that robots are likely worse, because they lack any capacity at all for questioning or for remorse, and they also allow great physical detachment from the horrors they can cause. They don't stop to eat or sleep either, and they'll certainly do things that can cause their own destruction.

Humans also may be even likelier to be loyal than moral if a drone with a gun follows them around to keep them on-side.

Humans have a natural aversion to killing, and yes, during war mentioned atrocities do happen, but the soldiers are not perfect killing machines. Often they miss intentionally and just shoot in the general vicinity of the opponent.

See this reddit thread: https://old.reddit.com/r/history/comments/b6k528/percentage_...

Troops can also desert which is especially prevalent in civil war situations. That's why during war, you need propagandists coming up with constant stories about how horrible and bad the other side is, so that you get people to shoot.

The unit that so brutally killed the tianmen protesters was one of the most brutal ones the Chinese military had. That's why they had to wait for days for it to arrive. Had they deployed different units, they might have fraternized with the protesters and joined them in their protest, making the task even harder for the leadership. Try getting even the most brutal unit to shoot at people in their own uniform.

For military parades, you can't afford 5% of people who intentionally don't march in lock step. That's why usually the troops which participate in such parades are isolated and fed propaganda leading up to the parade.

The human component is a constant variable that needs to be accounted for in military leadership.

>> For military parades, you can't afford 5% of people who intentionally don't march in lock step. That's why usually the troops which participate in such parades are isolated and fed propaganda leading up to the parade.

What? I've been on countless parades and don't remember any pre-parade propaganda. I've never even heard of such a think in a western military. Parades are just a regular day at the office. Honor guards and flag parties certainly practice heavily, but the idea that anyone "intentionally" doesn't march properly is ... odd.

Also, "lockstep" should not be used in reference to parades. The lockstep was a punishment once inflicted on convicts in an prisons. If you go back hundreds of year there was a style of marching, but it was nothing like the more recent method used as a punishment.

https://en.wikipedia.org/wiki/Lockstep

Sorry instead of Lockstep I meant the german term of Gleichschritt. At least in the GDR they infused the people who marched at the parades with propaganda for days before the parade. Yes, didn't mean a western country.

> The only real safeguard is layer upon layer of checks

But this is one such check. As I remember it it was at work in the fall of the Soviet Union, and it was worked around at Tiananmen Square (where they brought in troops from the country, not trusting the ones on site to be brutal enough).

> troops will mutiny if told to destroy their home country for said general's benefit

I’d generalise this check to all leaders. No man rules alone. Power only exists in its relationship with people.

Robotic armies permit force projection without any people. That changes the political calculus in all sections of society, from military leadership to billionaires to members of a terrorist group.

This works better if a large enough portion of the soldiers are conscripts (possibly rotated in and out) and not career soldiers. The conscripts know they will go back to their normal life, day job and be part of the civilian society again.

If it's just career soldiers, then it's in their career interest to enlarge the military's influence on society, increase budgets etc.

It just has to be a war crime to use these.

A domestic army of robots is really just a foreign army on contract. They have zero loyalty to anyone who doesn't directly control them.

This is true. Killer robots push societies in the direction of being more hierarchial, for the reasons you state.

This generalizes to automation of nonlethal means of population control -- crowd control weapons, surveillance and censorship. On the margin it'd hold for partial automation short of AGI -- reducing the needed size of the coalition on top.

How is this different from nuclear weapons?

If a general lives in the penthouse at the top of a skyscraper, and maybe has a few thousand people who support him, nukes are not going to help deal with revolutionaries on the bottom floor. They are too indiscriminate and can't be used against close up targets.

Whereas a general with a drone swarm and a few thousand supporters might be able to suppress a city. Fortunately a little far fetched, but what was impossible with nukes is only impractical with drones.

This is well put. There is also the idea that as armies got MI-24s/AH-64s, etc, opposing forces got Stingers/Iglas. Perhaps it will play out in a way where once armies get drones, OpFor gets counters - jamming / hacking / EM / whatever.

The land to be ruled by said rogue general will not be a radioactive wasteland.

Sufficiently advanced robots would leave a lot more of value behind than a nuke which makes deploying them much less costly to the person that wants to actually control the territory.

Seeing AI weapons as a weapon of mass destruction is a trending discourse.

It's a much harder weapon to control via regulation and treaties than nuclear weapons: there is no nuclear material required to hook guns up to computers en masse in factories, let alone improvisationally. Once this cat is really out of the bag, everyone who wants it will have it.

Manufacturing chips is an extremely complicated process and requires thousands of specialists from multiple different disciplines to cooperate. Building a new fab costs billions and the buildings are extremely sensitive for damage.

I think one of the reasons why the west wants to block mainland China from manufacturing chips is to prevent them from building a robot army.

It's true that building the chips is hard, but making it hard to buy general purpose small computers, or digital camera sensors suitable for vision, or rotor blades, or batteries, or antennas, is also going to be very difficult. Proliferation has already happened.

Maybe cheap LIDAR will be something we can keep a lid on? But regular vision is still likely pretty good for lining up a shot or finding a face.

> but making it hard to buy general purpose small computers [...] is also going to be very difficult

Just ban general purpose computers that allow running arbitrary unsigned code. I'm sure lots of governments and corporations would be happy to get on board that one, and many end-users wouldn't notice.

A cure worse than the disease. Also wouldn't stop the military from building up a drone army and then someone within it choosing to go rogue and take it with them. It only ensures that no one else can stop them.

I think the parent post agreed with you and was just being facetious -- 'just ban general purpose computing' isn't very likely to be a popular sentiment on hn!

I hope so as well ;)

In a war situation, likely supply chains will be disrupted, so the regime will have to use whatever it has in the country. There are chips in circulation, but each different chip needs different drivers, has different performance characteristics, etc. E.g. I read that Tesla changed a bunch of chips due to the current shortage and they had to write new drivers for them. That's one chip they had to replace.

Of course you could make your population turn over their old smartphones, remove IR filtering from the camera lenses and solder them to a robot body. But this will slow down manufacturing as it isn't well suited to mass manufacturing processes. Furthermore, it's a limited resource so you might have to deal with attempts by your enemy to exhaust it. Last, you'd be limited in fanciness, as in your robots can only process limited amounts. The best place to source the computation needed for really "smart" robot soldiers would probably be self driving cars as they have the AI processing power needed.

Sadly you can't make a general-purpose chip that only refuses to perform military tasks and otherwise functions normally.

The world made it through the cold war without major disaster, but it is easy to imagine history taking a different path, so I don't think that this is a comforting comparison.

Also, nuclear weapons can't be used 'a little bit' but killer robots can be -- it is much less clear where we draw the line. It seems likely that their use will be normalized/entrenched before we figure that out.

Though I hope that you're right.

One good zero-day could give total military and industrial control of multiple superpower nations to a single talented sociopath

Reply on news.ycombinator.com