Google Engineers Refused to Build Security Tool to Win Military Contracts

Problematic here is that I might not be aware of all my moral limits until I encounter a situation they might be relevant to.

Picture this: we're deep in the Cold War era, and Ronny Raygun is the president. The Star Wars initiative is in full swing. Nuclear mishaps and near-misses are still an all-too-frequent occurrence. Nena's red balloons are ready to fly. The President has just gone on the air and promised to "Outlaw Russia," and that "The bombing starts in five minutes." Some officers from the Pentagon approach your company's management and ask them to bid on a project that will involve launching dozens of military satellites equipped with atomic clocks, designed to locate any person, place, or thing on Earth at meter-scale accuracy.

Do you threaten to quit the company, lecture everyone who will listen about the potential for misuse of such dangerous technology... and miss out on the chance to contribute to GPS?

Or, how about a few years earlier, when they ask you to work on a colossal air-defense computer system using technology that hasn't even been invented yet. They want to use computers to vector fighters to intercept approaching Soviet bombers, and to target missiles to kill the ones the fighters miss. It's obvious to anyone not in uniform that the whole idea is insanely stupid, wasteful, and destabilizing. They want to spend $50 billion to develop a system that guarantees one whole 'nine' of uptime, that can be trivially jammed or disrupted, and that will literally be obsolete the day they turn it on since it can't deal with ballistic missiles.

So you tell them, "No, I don't want to work on SAGE. Peace out and fuck off." As it turns out, the system is indeed useless at its stated purpose. The spinoffs, on the other hand, not only revolutionize the entire IT industry, but create it. The next thirty years' worth of progress in computer architecture, memory and storage, networking, display and human interaction, and fault-tolerance is traceable directly to this boondoggle. All of these things would probably have taken decades longer to evolve without military funding and backing. Nobody dies as a result, but a bunch of your coworkers do get to play with a lot of cool toys.

Then there's the whole ARPAnet thing, funded by DoD to survive nuclear holocaust, asteroid strikes, plague, fire, and famine... any threat imaginable by anyone other than telecom-industry lobbyists.

Even the very wise cannot see all ends, and that goes for modern-day hippies too.

Working on missile guidance? Then I can see your point of view... but everything else in some way facilitates or contributes to facilitating such things.

One can rapidly reduce it to "fixing a Linux bug... facilitates the military industrial complex". You might wish to not work on things that directly are used by the military, but the vast majority of technology is indirectly used because the military and their supporting companies, processes, logistics is so broad that it's almost everything.

At which point can you say your work truly does not facilitate or contribute?

Why do you think it's more moral/ethical to abstain of supporting military? Maybe you won't do this but be sure that in some other country some similarly talented people will.

If you (and many others) abstain from it on similar basis you and closest people you know may be in danger of being affected by more advanced military equipment other (possibly hostile) armies can get.

Maybe it's just not in people heads that world and homeland safety is not given for free and granted forever.

If you don't want to be a part of it, good for you. But those freedoms didn't come cheap.

So what stops me, as a Google customer (which I am), to stop doing business with them, given that I see proof after proof that they are working against my interests?

This is the same problem as with the Snowden leaks. I don't care if Americans feel more safe with the NSA secretly watching their emails via gag orders, if my privacy as a non-US citizen is being violated, therefore at this point absolutely all US companies are suspects, starting with a negative ranking in my book; at this point I always prefer European ones. Also the usual arguments that countries collaborate with each other is bullshit, because in the EU I can sue companies, or my government, for cheap too, whereas in the US I have no rights.

And if you find that acceptable, then maybe balkanizing the Internet isn't such a bad thing after all, lets start with ICANN.

> There's also a real chance (though perhaps small) that an industry-wide refusal to cooperate with the US military could have disastrous consequences for the world

There's also a real chance that I might get hit by a brick and die tomorrow, yet that doesn't stop me from going out.

If you don't have an assessment done by experts of what might actually happen, then such arguments are bullshit.

Why do these workers have such a lien, whereas the people who happened to go into a career that isn't useful for the military don't? That sounds quite unfair. I'd say they pay their obligation the same way everybody else does: through taxes. And people who benefited more from that environment will earn more, and subsequently pay more as well.

That's probably true for US citizens, but I highly doubt non-US folks share that view (at least from a western European perspective I'd say definitely not).

wow. words fail me. I used to see the US as the "big partner overseas". That was 20 years ago. Maybe I'm projecting my own views and development here, but IMHO the picture the US as friendly or even just benign faded over a decade or so, then got utterly destroyed by whatever the heck US politics is today.

No. Education ideally is a gift from society to the individual, in the hope that it will be reciprocated, by work, but also by making informed decisions. And those can easily include refusing to work on a project (military or not) if the individual deems it harmful. Harmful to who? To the society that financed the education, to their family, or to "the world"? That's up to the individual to decide, too.

For example I can argue that a strong military is important to defend the country (or maybe even the West or whatever) while a military made out of drones and robots mixed with AI, driven by a Space Force is too easily deployed in combat, since risk on own loss is low, while the opponent might see civilian casualties, causing more harm than good.

The people of Afghanistan have a long and justifiable tradition of hating the British, mainly thanks to the three Anglo-Afghan wars of 1839, 1878 and 1919. Since the American-led invasion of Afghanistan in 2001, we seem almost quaint by comparison; our brutal imperialism is tinged with a vague hint of nostalgia for the days before cluster bombs and counter-insurgency tactics.

We invaded Iran during the first world war, backed a military coup of their government and ruthlessly looted the country for oil, but all of that seems like water under the bridge compared to the CIA's persistent efforts to destabilise Iranian society.

Pakistanis have every reason to hate the British - we imposed a bloody and brutal regime across the Indian subcontinent, then so badly botched the transition to self-governance that at least a million people died. Thanks to a decade of poorly-targeted and possibly illegal drone strikes by the US, nearly a century of totalitarian rule under the British is almost forgotten.

I can't remember the last time I saw a British flag being burned by an angry mob. Your government's decision to remain constantly at war probably hasn't done a great deal for your safety, but it's really distracted attention from our barbarism.

Then make a case, don't just state that you can and move on as if you did. For example here is my counter argument:

The US is currently stealing children, this is obviously not in the best interests of anyone, womp womp.

"a majority of citizens of democratic countries across the world"

Democratic countries like to keep their democracy not be ruled by a tyrant. The US Military is very tyrannical, the CIA has been a massive destabilising force in a lot of democratic countries.

"Given that society created the environment"

The tech companies created their environment, just like their employees spent their time and effort acquiring their skills. You are claiming authority over other peoples work with no justification.

"I'm not in favor of compelling anyone to work for any purpose"

You just stated the opposite. These are hypocritical weasel words so you can walk back your argument.

"There's also a real chance -sic- refusal to cooperate -sic- could have disastrous consequences for the world"

Such as? You make a lot of wild claims with absolutely nothing to back it up.

Your argument is ultimately trying to make a case for totalitarianism as long as the US are the totalitarians.

The general reputation of private sector weapons makers used to be very low. They were "merchants of death." This comment assumes their legitimacy.

The USA was formed on a basic distrust of the government, including checks & balances, and non-federal armament in the 2nd amendment. It is not allowed to be a single, trusted federal power, but rather one whose powers are defined by limited and explicit allowance, with default powers resting on the distributed states and people.

The notion that the USA should stand in a singular trusted and empowered situation over the rest of the world neither fits with fundamental American concepts, nor notions of robustness in general. Strength of militaries should be distributed among independent allies, not focused on any single point of failure or corruption.

The military strength of the USA certainly benefits the USA. But a singular point of above-and-beyond military strength is not a model of benefiting the world.

Given that the U.S. is being consistently voted as being the biggest threat to world peace, I'd vaguer not. That does not mean we want Russia or China to dominate us instead, it just means the world doesn't want to be bullied by the U.S. either. You do realize that every single empire in history assumed it has a more moral position to bully others that those that came before it, don't you?

I don't know how it reflects in American society, but in the rest of the world the fact that the U.S. killed a million Iraqis here, a few hundred thousand Afghanis elsewhere, enabled the slave-markets of Libya, funded terrorists in Syria and is currently helping to commit genocide in Yemen does not reflect anywhere near great on the U.S.

Some of us actually think that the people from these places are also valid, genuine human beings who deserve to live, so no, many people do not think the U.S. has any moral high ground here. It may be hard to swallow for some U.S. patriots, but I'd say you should be proud of the country and what it is supposed to stand for, rather than being proud of the government, (government != country), and what it is doing in many places.

Despite what many Americans seem to think, you do not have the best healthcare in the world, nor the best education system, nor the best infrastructure and I could go on and on - please focus on that instead of empire building.

One point I haven't seen raised here is that "a strong military" is not a one-dimensional proposition. The military does a lot of things in a lot of contexts, and they don't all have the same moral value.

If someone wants to build better Humvee armor for the US Army, I wouldn't criticize them at all. I don't think there's a reasonable case that better-armored Humvees are worse for the world. (Not least because we proved in 2003 that a lack of proper armor doesn't deter the US from starting wars.)

But I think the work Google engineers are likely to do for the US military is not likely to advance Pax Americana, which is mostly about a nuclear umbrella and secure international shipping lanes. Instead, it's very likely to abet either the drone war or mass surveillance, since those are closer to Google's area of expertise.

Quite a lot of people who are not strictly against US superpower status are still deeply opposed to the specific sorts of action Google would be best-placed to enable. And since they haven't actually signed on with the military, it seems totally reasonable that they would object to doing such work even if they do value a strong US military in other domains.

Which society does have that lien, the one I left behind that fully payed for my education (although almost everything I know about computers I learned on my own) or the one hosting the company I happen to work for right now? What about some years in the future when I move to another company/country?

Yes. And that's the rub. I had the good fortune to work in research for the military for almost 10 years. They have more patience and willingness to adapt than any commercial company I've known. Their time horizon spans a much longer interval than most corporations. Did neural networks out of the PDP groups work in the '80s, and when things were too slow, they went yup and bought a bunch of Mercury computer cards to speed stuff up without squawking or delaying. Now I do work in stereoscopic machine vision. I'm pretty certain that biological or machine, the lowest levels of the visual stack are just targeting systems based on feedback loops. Here I agree with Dr. Russell at Bezerkely. We must not, can not, make autonomous killing machines because the consequences are potentially too catastrophic. So I refuse to work on anything resembling autonomous mobile weapons systems, as do many many others in this field. This has happened before - we were playing with enhanced radiation weapons in the '70s and '80s (neutron bomb), and over time pretty much every participant decided it was a very bad horrible no good idea. I would argue that for talented players capable of dragging the future into the present, the issue is not that the military is bad, it's whether a specific application they want is unequivocally bad. It's not about who, it's about what.

If you ever played boardgames (especially German style) maybe you'll remember some games when only one player developed some resource unchecked by the other ones. This usually leads to an easy win for that player. Not easy to notice it in the earlier turns but then the differential in income (whatever it is in that game) is so obvious that everybody understands the game is over well before the end.

It may be the case these people are all friends or acquaintances and enjoy the opportunity to work with each other.

That isn't enough. Due to the frankly idiotic way corporate investors think 'any company that isn't growing must be dying'. Google will need to continue to demonstrate growth forever despite that being impossible. Growing in the ad market is hard because they're already the biggest player, so Google are looking at new things to do. This is just one of them.

Where the balance tips and they become complicit is when they offer customized and specialized solutions that further a particular mission. Here I think good engineers will draw the line as a matter of conscience.

Why bother with any new products when search and ads are extremely profitable?

I think it makes sense to look at this from the perspective of Google Cloud since I imagine that's where the contracts are. Google Cloud isn't a leader in this space. In cloud, having more customers is a competitive advantage both because of marketing and also because of technology/cost.

With more users, you can share costs more and have better forecasting since any given customer's specific usage will should not affect your total usage assuming customers are fairly similar in size (not true in reality..). Then hopefully you can run closer to capacity, lower prices, and then gain even more customers.

> Given the obvious conflicts with their company culture (obvious to rank and file employees, anyway) it doesn’t seem like a very appealing trade-off.

I don't agree with this to be honest. At least, I certainly wouldn't consider it "obvious". I'm a rank and file employee and I was surprised at the outrage. I still don't really see anything wrong with supporting the government or military but I can understand how this makes some people uncomfortable, especially if they are not US citizens.

They're a very long way from there, see for example this: http://defence-blog.com/army/combat-tests-syria-brought-ligh... (which also happens to be my latest separate post submission). People tend to over-estimate the technical prowess of the Russians, there's only so much that the military of a country with ~160 million people can do to technically compete against both the US and the EU.

Many promising young researchers from around the world earn their stripes at US universities, and loads of them would be happy to stay and become Americans. Unfortunately, the US immigration system makes that difficult and lots of them get sent home.

The best part about that plan is they don't even have to develop their own AI weapons. They could deny that capability to adversaries just by allowing those workers to take up interesting & lucrative jobs in the American civilian economy.

https://www.bloomberg.com/view/articles/2018-05-31/u-s-shoul...

The value of (relative) freedom made the US powerful. If the US becomes just like the bad parts of China or Russia, who cares of China or Russia come out on top instead of the US? They will have already won the war of ideas.

>Any data you give to a company, even if it is ethical now, will be used against you in the future.

https://www.theverge.com/2018/6/21/17488328/microsoft-ice-em...

I doubt many expat H1B's from the subcontinent will have any qualms about military work.

If you were to build AI technology to do this, you want it to be as incompetent as possible, so the military doesn't use it.

AI wishful thinking is not helpful.

This article certainly raises that eyebrow. It stipulates the problem back then was a feature called "airgap" which (in the normal meaning of the word) is rather trivial and usually is for defense and not offense.

But to me it all reads like a per-usual politics BS story to discredit a group and downplay a problem.

I certainly wouldn't want to live under (in?) ISIS, but that's because I had a luck of being born in a stable country where the government doesn't hate me and doesn't treat me as an underclass, where the police doesn't rob me or beat me,...

Many other people had no such luck, and I will not simply say that they deserve to die, just because someone slaps some label on them without any obvious nuance.

Usually goes along with the air gapped system running in a small intranet you strictly control access to.

Its not a bleeding edge technology as the article says (maybe the author made a joke?) but a standard security measure to protect highly sensitive or critical systems.

You seem to think everyone on HN agrees with each other.

If everybody had the same point of view, there would be no ground for a discussion...

We need more media literacy around HN and Reddit. You just need a few users to upvote something, and people often respond when they see the opposing view (this is why conservatives and liberals in the US so often take the most extreme statements on the other side, and critique/retweet them; better engagement and an easier way to caricature the opposing side).

It's interesting to think about how different posts attract wholly different tribes, meaning HN is hundreds of sub-groups not a monolith. There might be a post on a Ruby gem, and the entire commenter group are Ruby-ists who hold common views on opposing languages. You might see this as "HN believes X", but instead it's the subset of people attracted by that post that you're really interacting with.

Will add this as an example to my media literacy guide for engs:

https://github.com/nemild/hack-an-engineer

What kind of logic is that?

We don't necessarily need to match lock-step with terrible regimes to ensure our own security, and our own security is the only argument for matching them in the military field.

For example, if someone else uses gas or landmines, we can use masks and airplanes to defeat them, without having to resort to those things that can't pick their targets, and remain on the field, where children play, decades after the war ends.

In the physical version of total war, you can only provide defense with military tools. You need military anti-aircraft guns to shoot down military aircraft. You need tanks and artillery to stop enemy tanks and artillery.

To provide defense against cyber total warfare, you don't necessarily need military technology. You just need secure civilian technology. Google is already doing a lot of work in that area. Examples include Project Zero to find and close security vulnerabilities; their work to normalize, monitor, and enforce proper use of TLS across the Internet; Chrome browser and OS security work; Gmail security work, etc.

Government institutions can provide predictive surveillance that Google cannot. Only the Cyber Command, CIA, NSA, DIA, etc., and their international partners, have the ability to see inside the HQs of potential enemies. And those are the only agencies permitted to act in an offensive cyber capacity.

Is Google under any obligation to support that work? I don't think so. Google can do, and is already doing, A LOT to help the U.S. defend against cyber warfare.

We have an obligation to stand up for our beliefs if we are going to see the change we need.

By most accounts, the USA in the next war will be the bad guys.