Don't Get Distracted (2017)
calebhearth.com
calebhearth.com
But the company is a major defense contractor and it was very unusual for them to be involved in humanitarian work. So it made sense to question the requirements after a less generous reading. Detecting mostly occluded humans in a highly cluttered and destroyed urban environment has other use cases. They got as far as "highlighting the survivors with a laser designator" before I started to ask some very pointed questions and start discussing the ethical considerations of us helping to build a system to kill people.
I was asked to leave.
After that meeting the potential collaboration was rejected and it never came up again. The company continued shopping around other places to get help building their "earthquake rescue system".
Never get distracted. Always question the purpose of work. Consider ethics.
Getting another job can be a pain. Living with knowing what you helped to build would be much harder.
The handful of US based defense contractors I've had contact with were VERY up front about what they develop. I think they more so wanted you to know so everyone was on the same page.
Lockheed Martin doesn't need to trick anyone into working for them, and that includes software developers.
If this was a real thing that actually happened, everyone would need a security clearance anyway.
That matches my couple data points.
I did a lot of software engineering consulting work through a Federal contractor. When I was first approached, the director told me some of their customers were military, and how would I feel about that. I said something generally favorable, and that I only had a personal rule that I didn't want to work on weapon systems, nor domestic surveillance. That was fine with them, and we did years of great work, on very positive safety programs.
Earlier, in school, I did a final project for a class (a simple "agent" system to help people find online communities of interest in Internet chat). The professor later told me, out of the blue, that a federal contractor wanted to license my little project(!), for surveillance(!), and they'd pay me to advise. At least they were upfront about what they wanted it for. I declined and discouraged them.[1]
[1] One reason was that I turned off by surveillance of the Internet, as a pre-Web Internet kid (also raised on anti-Soviet/anti-police-state propaganda) who still thought of the Internet as a better world that we insiders wanted to bring to everyone else. The other reason was my little technical approach would've been a waste of money for that surveillance problem. IIRC, I was polite in declining, and didn't mention the first reason, but was probably impolitic in telling them the latter reason.
At least as far as the US goes I've never heard of them awkwardly trying to get people to develop something for them in the manner described. In my experience they're very up front about what they do and aren't short on people willing to work for them.
Not to say the story was a lie, I don't know, but it sounds very unusual.
I didn't disbelieve the article's story. It might've just, for example, been someone's pet project, with only budget to hire some co-op students and an inexperienced new-grad coder, to make a demo, to pitch for a real project/contract. Or to proof-of-concept a method, to possibly be properly designed and implemented by a real engineering team. Maybe they didn't think they needed experienced serious engineers with clearance for this compartmentalized exercise.
Another, maybe less-likely, possibility was something involving a cleanroom implementation as demonstration it was "obvious to first-year students". Or to find a plausibly alternate cleanroom implementation, by people who hadn't been tainted by knowledge of the encumbered one. Guessing not, but I did see something like this done at least once. I'd bumped into this affable young high school student, who was idling around in common areas of my lab, and was outgoing enough to strike up a conversation with random passerby me, and start telling me he was working on a project. It turned out a professor had tasked him to reinvent this one algorithm that was some alum grad student's thesis, which had been patented, and which the alum had been milking rather hard... and the professor didn't tell the high school student about the prior work.
I also once had a current PhD student hired as an intern, to apply very niche expertise towards a speculative new feature ("wouldn't it be great if we could do this cutting-edge thing in the new system"), and they were mostly successful, but the product ended up going a different direction. (The PhD student did OK: they then had understanding of a great, tricky real-world domain example, to guide research. A lot of research uses over-simplified examples of applications that they never had the opportunity to really understand.)
I did have a twinge while reading the article, wondering why it seemed like they were about to go into methods used by what sounded to my naive ear like a tracking/targeting component of a military system, but I would guess that they didn't spill any secret sauce beans.
I would love to have an honest, thoughtful discussion about how a war can be prosecuted between two powers with minimal civilian harm, but it's not possible when people aren't even honest about what constitutes murder and what doesn't.
From your post, I think you’re starting from the pov that governments have the right to decide what is murder vs an acceptable killing. Some of the people I’ve met who are most interested in these ideas are staunch pacifists with a strong religious motivation (e.g. Quaker or Methodist) and reject the idea that governments can declare any killing to be acceptable. I don’t believe they’re dishonest, for all that it’s a very different starting point.
Many of us, including many intelligent people, think that all such premeditated killing is objectively murder regardless of the political context.
Please don't handwave this fact away.
Some people feel even stronger than I do and would prefer to avoid being involved in killing even an enemy combatant.
Discussing 1-how to limit civilian casualties during an otherwise legitimate conflict between two nations is one thing, 2-whether those civilian casualties constitute murder (and 3-by whom) is something else. But through pretty much all of human history, nearly all people have recognized a difference between killing during a war, including innocent civilian deaths, and intentional murder. If you want to discuss #1 you had probably agree on #2 first, don't you think?
At least in the US, we are not a direct democracy so it's not important or even an desirable to have everyone decide where their money goes based on their own individual morals or ideology. We elect leaders and in a perfect world they're smart enough (lol) to see that taxpayer funds are allocated in a manner that best helps Americans (lol).
Less snarkily: even if we agreed that paying taxes makes one complicit on evil acts done with that money, it doesn't mean that person should not avoid being complicit in other evil actions where they can.
But it turns out, authoritarian states wanted to buy the very same tech to locate undesirable people, record their associates and travel history, geofence them, etc. Also turns out salespeople gonna sell. And that's how you go from an ethical position to another kind. People did quit over that.
The obvious military applications for this technology are explored in Tom Clancy's novel "Rainbow Six" (1998).
That said, working for a US military contractor while not being supportive of the US military mission is kind of a silly thing to do.
Wasn't a lot of the Snowden stuff that tools intended to be used on foreign enemies would be used at home?
It's interesting to me that many people who work for giant tech companies that carry out indiscriminate data collection feel confident in their employer's internal processes and controls, but don't extend that same optimism when it's the government that pinky-swears they aren't going to peek at anything they shouldn't.
Nope
> The NSA has built a surveillance network that has the capacity to reach roughly 75% of all U.S. Internet traffic.
> An internal NSA audit from May 2012 identified 2776 incidents i.e. violations of the rules or court orders for surveillance of Americans and foreign targets in the U.S. in the period from April 2011 through March 2012, while U.S. officials stressed that any mistakes are not intentional.
> The FISA Court that is supposed to provide critical oversight of the U.S. government's vast spying programs has limited ability to do so and it must trust the government to report when it improperly spies on Americans.
> A legal opinion declassified on August 21, 2013, revealed that the NSA intercepted for three years as many as 56,000 electronic communications a year of Americans not suspected of having links to terrorism, before FISA court that oversees surveillance found the operation unconstitutional in 2011.
https://en.m.wikipedia.org/wiki/2010s_global_surveillance_di...
The kicker to me:
> it must trust the government to report when it improperly spies on Americans
The courts told them to watch themselves and to self incriminate if they do crimez. An honor system. Which naturally, they did not snitch on themselves to the courts, because why would they.
Also let's not forget that they can make all the illegal surveillance legal in an instant just by labeling any person a "terrorist"
https://www.vice.com/en/article/google-fired-dozens-for-data...
https://www.independent.co.uk/tech/amazon-ring-camera-spying...
There is no similar venue for oversight and accountability over the tech industry, which almost certainly holds more sensitive data on the American public.
There's a reason the things they do are done in places where US laws don't apply...
> CIA black sites systematically employed torture in the form of "enhanced interrogation techniques" of detainees, most of whom had been illegally abducted and forcibly transferred. Known locations included Afghanistan, Lithuania, Morocco, Poland, Romania, and Thailand
https://en.wikipedia.org/wiki/CIA_black_sites
Then there's GITMO
They broke major laws and tortured people, hacked the computers of the congressional oversight group to delete evidence, got caught hacking, lied about it, and nothing happened anyway.
The mental model you have where the CIA is in any way constrained by the law is an utter fiction. They would summarily execute Snowden in a millisecond if they could.
This is who is lurking behind the curtain, petty, fallible humans just like everywhere else: https://www.newyorker.com/magazine/2022/06/13/the-surreal-ca...
Don't make assumptions and don't put words in my mouth
You never read about LOVEINT?
This kind of thing is routine in the tech world, too: https://nymag.com/intelligencer/2010/09/elite_google_enginee...
In my youth I applied for a defense force job in my country. I was turned down because I didn't have recent experience in the exact skill set they wanted for the first project (iirc it was Visual Basic front-end UI coding, lols. My employer at the time said they'd made a stupid mistake, I should have appreciated this statement more at the time) but it made me think. I never went for another job in defense. Honestly I don't have a problem making machines that might kill (such things are, sadly, often necessary) but on reflection, I sure do have a problem with killing someone that I don't agree should be killed.
This one time I did help a friend fix a machine that was cleaning the defense force's boats, but that's another story.
Many reasonable people support the stated mission. But that shit ain't the truth.
(with apologies to Arthur C. Clarke)
They get into the interesting problems and miss the forest for the tree.
Plus it's fun to be among smart peers, and well-paid.
The author woke up from this, but many do it for their whole life and see no problem with it.
That's why I rejected interviews from Google and Facebook. Because I knew I would do the same. It's too tempting.
Copious quantities of impression data? Get to do big-data-ish things? Play with things like Cassandra etc (back then kinda interesting), k-means clustering and fun stats stuff and eventually ML?, high throughput & low latency transaction processing... shaving milliseconds off here and there... and lots of money in the sector... to hire devs, to do these things... which were at least back then somewhat intellectually challenging...
Especially for people not in the Bay Area, opportunities like this were/are hard to come by...
But then once you take a step back and look at how the sausage is really made, you start to feel icky.
Granted, that was ad-tech in 2010ish time-frame. It became much much worse.
Luckily within Google I was able to transfer out of ads, and into things that seemed on the surface much less icky (consumer hardware, etc.) But the thing is... no matter what you're doing at Google, that project is funded by ad-tech... So...
But seriously, I think there's a big difference between ad-tech in the strict sense of pre-"social media" ad tech and ad-tech now. When I was doing it, behavioural targeting was just getting started. One of the startups I worked at was trying to get knee deep into it (pretty incompetently, IMHO), but overall it was still "here's a stream of mostly undifferentiated impressions/bid-requests/clicks" and that was that...
Overall, it was hard to make a strong moral judgement at that point. Ads paid for the Internet as we knew it and most publishers could not survive without it.
But after about the Facebook IPO, we're talking about a very different thing, one that gets more much more morally grey-area, and sometimes outright just evil.
If the thing you're working on ends up with pushing even just one teenage girl into an eating disorder... How do you feel now about it now, even if you dumped your earnings into charity?
Think of it like this: what if I tried to argue with you “think of all the good we can do with the profits from the baby-pulping machine!” as an executive of the baby pulping company. Maybe I run a dog adoption agency with the profits, or a soup kitchen. I’m using acts of charity as a way to launder the horrific actions of my company.
Almost everything we do as a consumer or worker in a capitalist market is tied somewhere down the line to a generally-agreed undesirable outcome somehow.
So the really tricky part that involves using your brain and heart is figuring out where that line is.
(And it's sad that in large part our voice in the world's operations is reduced to mostly ineffective "buy or not buy" ... )
In the early 2000s, I wrote software for a startup where the founders and initial equity-holders had pledged 10% of the founding stock to a certain charity. This was part of the marketing approach for the company- help us succeed and it benefits others (without affecting the competitiveness of our cost/pricing structure of our services in any particular tangible way.)
There were some weaknesses in the model of mixing charity with startup-mindset (the model later changed to 10% of profits and we actually brought the charity work in-house eventually), but as a lead engineer, I actually found it quite helpful in pushing back on questionable (in my eyes) things the CEO was otherwise tempted to pursue and asking me to build (the CEO not being initially a technical guy and having any developed sense of technical ethics around spam, dark web practices, etc). If we did something really unethical (and were caught), it would actually hurt the charity we were also all trying to help and undermine part of our company's core differentiation.
If you are a technical founder and bake this sort of charity commitment into your startup and its marketing on day 1, you can help people at all levels of the firm develop a more ethical culture than the average startup which faces all sorts of (often stupid) temptations.
* If you're unsure, quit
* It's fine to distrust the system, and as with most of the population, do something else
* If you do work on a Serious Problem, decide what your line in the sand is, and ensure that aligns with the organizations you work for & with
* Constantly reevaluate yourself, the people, and the work
Extremes like "don't do things if they can ever go wrong" would suggest the only answer is to do nothing. That doesn't work for most things. Likewise, I don't like painting people as unethical just because they have evolved different belief systems or, quite often, a more nuanced & evolved view of something difficult. Thankfully, I find most people in the US, and in many other countries, to try to do the right thing on average. This is even more true of people grinding on Serious Problems. So it often comes down to personal responsibility and awareness for whatever you care about, keeping the alignment with those around you... and leaving when that stops working.
The AI & robotics booms have made this stuff even more important. We view it as a chance to do good, and with more accountability & quality than when those with less scruples & ability do so. But agreed, we pick where we work, and ensure new staff agree on where we drew our own lines in the sand.
e.g. I work fulltime in Rust, and love it. I look at the job postings that emphasize Rust ... and 80% of them are crypto/web3 companies. I won't work for those companies because of my personal beliefs about "crypto"...
But if I were to lose my pretty decent job right now... and the bills started to pile up and my family was in financial danger... The argument becomes more difficult.
I didn't realize that the video embed was broken until someone pointed out that this was making the rounds again, so I've just fixed that. If you've already looked, the video is now available. The content is more or less the same as the written version so don't feel like you need to watch it if you've already read the post.
What is different about that situation from the situation you described in your talk?
What factors would have to change in the global balance of power for you to consider building systems that kill for American companies?
I won't take the bait on defining the differences between wartime and peacetime work.
However, it describes a real scenario playing out literally as we speak.
While I absolutely don't mean this in a confrontational manner, if your ethical framework doesn't provide you with means to address a real-world ongoing situation, what's it even useful for?
The decades of American misadventure in the Middle East have been devastating for the future of world peace. In the quest to occupy two countries and engage in asymmetric warfare against relatively poorly equipped terrorists / freedom fighters the United States has burned a tremendous amount of social capital and soft power abroad. Additionally these prolonged conflicts have been hugely unpopular domestically and have had detrimental effects on the morale and functionality of the armed forces. People and institutions are burned out at the thought of supporting the armed forces, and your talk is a prime example of that.
This threatens world peace because even in times we consider to be peaceful authoritarian forces are plotting against democratic institutions. While America burned trillions of dollars of assets and social capital, Russia and China have been quietly amassing the resources to wage war and shatter the peace that people like us as well as our Ukrainian, Taiwanese, South Korean, and Japanese counterparts. At the same time they've been not so quietly waging war in a different domain and have built up a disconcertingly powerful fifth column through concerted social media campaigns that affect large sites and Hacker News alike.
I used to be very opposed to American hegemony and interventionism and for good reason too. What happened in Iraq and Afghanistan was an atrocity. Dick Cheney and others should be living out the rest of their lives in the ICC detention centre and the oligarchs who indirectly amassed their fortunes from these conflicts like Liz Cheney should be stripped of the resources and influence that they have in our society today. Unfortunately that's not the world we live in.
Instead we live in the reality where these people are free to run amok and their authoritarian counterparts in Russia and China are preparing for an all-out assault on global democratic institutions and individual freedom.
What's even more concerning is that we live in a world where the resources of the US military are depleted. The decades spent fighting insurgencies have left the military unfocused to address the rise of countries like Russia and China. The US can't even supply enough shells for Ukraine to wage war effectively. And there plans to rise to the level of production needed for that conflict and others like it are too little too late.
In the seeingly impending conflict with China over Taiwan the wargame scenarios paint a dire picture.[0]. America has insufficient stocks of missiles to wage a protracted war with China with many supplies estimated to be exhausted within a week of conflict and the lead time for producing replacements is measured in months to years. China comically outstrips the shipbuilding capacity of the United States with the US Navy so desperate to build naval ships that they've begun outsourcing production to South Korea[2] which again seems too little too late and precariously close to Chinese missiles.
If America loses access to the advanced production of South Korea[3] and Taiwan in the near future how will it ever scale up production to meet the rising threat of authoritarianism like it did in World War 2?[4] While the US has let their industrial capacity deteriorate and has meagre stockpiles for war China is not so quietly building theirs.[5] China dominates in the production of crucial commodities like steel, aluminum, copper as well as more advanced products like batteries and solar panels. They are constructing massive factories[6] to dominate the electric car industry that can easily be repurposed to producing drones which are proving to dominate the battlefield in Ukraine and Russia.
I agree with you completely. We must be ever mindful of actions and their effects both intentional and unintentional. There are consequences to action and inaction alike but question is not whether the price of action is high, but whether the world can afford the cost of inaction.
[0] https://selectcommitteeontheccp.house.gov/sites/evo-subsites...
[1] https://www.visualcapitalist.com/countries-dominate-global-s...
[2] https://www.navalnews.com/naval-news/2024/08/hanwha-ocean-be...
[3] https://www.visualcapitalist.com/which-countries-have-the-mo...
[4] https://www.construction-physics.com/p/how-to-build-300000-a...
The issue is the common practice of police asking crime victims if their assailant looked like any celebrity. If they give any celebrity's name, they put that celebrity's face into their FR system, and start harassing anybody in their FR system that looks like the celebrity. That is omnipresent, and a very good reason to alter your appearance if you are unlucky enough to look like anyone famous.
Even if there's nothing morally objectionable, like other places I've worked at, the product is so worthless that people would enter the same patterns of behavior.
> The unifying factor in all of the stories I’ve told is that a developer wrote the code that did these unethical or immoral things. As a profession, we have a superpower: we can make computers do things. We build tools, and ultimately some responsibility lies with us to think through how those tools will be used.
No, it does not. It still boggles my mind that people actually seem to have this model of morality. If everyone thought that way, the world would not have nuclear power, to name just the most obvious example that comes to mind.
Nobody is "responsible for" making it possible for others to do evil. That's their choice.
(To expand a bit, because clearly not everyone shares my intuition: if this sort of moral failing can attach at one degree of separation, logically nothing prevents it from attaching at arbitrarily many degrees of separation. If I become culpable because my software "is used to kill people", then any person or company who enables me to write the software faces the same judgment. Vim is used to write software that is used to kill people. Do you really want to put that on the guy[0] who tried to get you to donate to humanitarian aid in Uganda?)
(The same logic is used by activists to demand participation in various seemingly arbitrary corporate boycotts - and people I care about have been harassed because of it. The thing about "following the money" is that it goes literally everywhere.)
Both of these are perfectly valid lines of moral reasoning. Which one you choose is going to be a personal decision. Debating about which one is more "right" devolves into a philosophical discussion.
Does it "boggle your mind" that some people choose to be vegetarians?
I understood the article as reminding folks to actually think about what you might be facilitating, and make your choices accordingly.
It's a spectrum.
The code described at the beginning of the story locates objects, which are presumed to be in the vicinity of a person, whom the military might then kill, after presumably having used that location software without the victim's consent. By this standard, we should hold Tim Cook responsible for every case of stalking involving the use of an AirTag.
And um, yeah, the AirTag release was bad enough for obvious reasons and Apple had to make significant privacy changes. Almost as if they forgot to consider that their work could be used for purposes outside of what it was created for. Could've been safer from the beginning.
Edit: turns out Apple is being sued for Air Tags' uses in stalking [0]. So I'm pretty confused by your point since would presumably Cook care about Apple being sued. Will he personally be criminally liable? Doubt it. But it's not like he's blameless or ignorant to the situation.
https://apnews.com/article/apple-airtags-stalking-lawsuits-e...
Others are free to consider whatever they like about possible uses of my work.
They are not justified in blaming me for those uses, and I am especially unsympathetic to arguments that I ought to blame myself.
To take your same point to its logical conclusion, no one is responsible for evil aside from the one who pulls the trigger.
As an aside, one thing I notice a lot on internet forums is the tendency to immediately jump to these two extremes (often as a form of strawman). Might be projecting here, but I think it's an attempt to get internet points and seem smart, e.g. debate culture. Though I could totally see that maybe everybody can intuitively see these absurd conclusions, so it follows that there will probably be one genuinely disgruntled reader that that finally reaches their breaking point. I know I've certainly made similar comments.
How do we navigate this line? Ultimately I think the answer can only lie in human experiences, and thus I'm glad that the original article exists. It's another datapoint. (though this spawns a whole other discussion about how we get our data)
Yes, and my argument is that building those things doesn't have the "impact" referred to; using them does.
> To take your same point to its logical conclusion, no one is responsible for evil aside from the one who pulls the trigger.
Tools, by their nature, have explicitly designed uses, and potential cascading consequences of that use. This is one of the reasons that open-source licenses include a disclaimer of warranty: for the legal protection of the author, not just from claims by users, but claims by third parties injured by those users.
As far as culpability goes, I'm much happier drawing the line in a place where, if you keep applying the same logic you used originally, it will stay put rather than moving inexorably further. Contrary to what many have tried to tell me, intent does matter, a whole lot.
To an extent, I'm with you. Is a brick maker responsible if someone uses one of his bricks to commit a murder? Obviously not. On the other hand, if you work on, say, proximity triggers on missiles and never stop to wonder who your missiles will be used on, I'd say you've abdicated a core ethical responsibility.
I don't have a good answer on where the line is, and I've thought about it a lot.
This guy won't touch anything related to the military.
Palmer Luckey is part of society's "warrior class that is enthused and excited about enacting violence on others in pursuit of good aims" [1].
I'm somewhere in between.
1. https://techcrunch.com/2024/10/01/palmer-luckey-every-countr...
More substantively, my company is a contractor and has publicly committed to neither working on 'weapons' nor 'destructive financial systems.' Both terms are nebulous but we generally just decide en masse as a company (there are only 20 people) what does and doesn't count. I don't necessarily care for these particular lines in the sand but it's important to have some kind of process for determining and thinking about things, which I'm happy we do.
Someone is complaining about writing software that could be used in implementations which kill people, whereas men in these countries straight-up don camouflage uniforms, helmets, and armour. They are given rifles, and are taught how to shoot and kill people, and sometimes even get to kill people.
Cut out the middlemen and all the abstraction, I suppose; program in C and assembly rather than in Python.
I do wonder about this whole premise. Let's say your country is under attack by an oppressive neighbor or known bad terrorist organization?
Would people feel differently?
I get the same weapon could be used differently / against less bad targets, but it doesn't change that they can also be used to defend / attack people who genuinely want to do you harm.
I do not see this as a cut and dry call.
Then revenue growth stalled and we shifted to enterprise consulting, which caused a lot of the early employees to quit (including myself). A few months after I quit they took on the Maven project that Google and Amazon employees protested against and in no time after that they had an office near DC with a bunch of ex generals on their staff. Now they work on secret projects that nobody still working there can tell you anything about.
Moral of the story, you can be a well meaning person and work on cool tech but you have no control over how it will get used in the future. The same thing that happened in CV will happen with all of these LLM startups once growth stalls and they're unable to raise another up round to cover their huge expenses.
Still the example of getting focused on engineering problems and not what the code will be used for lands well.
This is something that only individuals can do.
And there will always be individuals willing to write this code, and that's not anything that can be solved by any number of Guilds either. The military will pay anything it takes to make it worth working outside of the Guild's rules.
Of course those associations are always a mixed bag, often fostering some mediocrity or being themselves corrupt.
But definitely feel like there's some aspect of that missing in our sector.
My local teacher's union has some affinity groups and one group decided to invite a speaker who in the past has had some very strong words for people of a particular religion and point of view. Needless to say it is a mess of a problem.
I'm not sure I'd work on a military project, but I also wouldn't be comfortable with a guild telling me that I or anyone else can't / shouldn't.
It is a complicated topic I think; on one hand lots of people don’t want to work in those types of applications. On the other, we have a decision-making process for how violence is applied. If we are going to start inserting more veto points in there (which I think is a good idea), I don’t know that “has some rare technical skill” would the first qualifying characteristic I’d look at, right?
I don't think there's a reasonable argument that e.g. Keurig engineers need to grapple with the ethical dilemmas of murderers and terrorists using their products for example.
Something like that but then applied to software engineers specifically
I mean, field of study changes perspective. Recently, there was that backlash against a release of a 1-mil post dataset of bluesky users by some people on bluesky. https://eugeneyan.com/writing/anti/
I personally didn't think it was a big deal because the API and dataset is already public. And also if your text isn't available to LLMs, then your values are effectively invisible to the future, as society depends more and more on LLMs. To me, I think it's better that you get your thoughts out there into LLMs. As more of society relies on LLMs, culture wars will be fought not just on the internet, but in the embedding space of LLMs. So, given that, would these people want me to decide for them?
I dunno. Just as journalism lost the trust of the public by deliberately taking an activist stance, so too would engineering lose public trust by deliberately taking an activist stance. (You may say tech already lost public trust, but I think of tech as the industry and engineering as the profession).
Go look at the latest "Ask HN" thread - there's two projects proudly trumpeting their ability to allow applicants to cheat on their interviews. On the FAQ of one of the apps, a common question apparently is if it will work during proctored exams.
https://news.ycombinator.com/item?id=42375440
https://news.ycombinator.com/item?id=42383928
With this kind of loose approach to morality, is it really any surprise that the world is filled with devs who build involuntarily telemetry, spyware, etc.?
What I can agree with is that placing a moral cost on engineering designing weapons in some way increases the actual labor cost of weapon R&D, I guess.
I applaud anyone who imbues their work with a sense of ethics and responsibility, but most tasks, inventions and creations arise in an environment filled with muddy grays, rather than being clearly good or evil.
Regardless of whether "we" want them, we already have them, and that genie isn't going back in the bottle. In my view, every person has to decide their ethics and then act accordingly. The alternative is forcing people to accord with the principles of the employer or government or guild or whoever. There just isn't a coherent supermajority morality anymore - and no real path to one.
I don't have a solid object level opinion on the llm stuff. For the most part it seems like a tempest in a teacup to me. Some things will get easier, some things will get more annoying, and everyone will still have to get up and put on underwear one leg at a time.
I don’t trust my phone on a fundamental level. I also have mentioned things out loud and gotten ads afterward about them.
On the one hand, those two statements absolutely don’t prove that my phone is listening. Just because I don’t trust my phone doesn’t mean that it’s being used maliciously, and the ads I’ve seen might just be a coincidence.
On the other hand, those two statements together are definitely suggestive. They feel suspicious. It would be so easy for Apple/Google/Samsung to secretly listen and make a boatload of money by telling me a lie. Lord knows they’ve lied about other things.
So yeah, balance of evidence is that phones probably don’t do that. But don’t be surprised when people suspect that they do.
Also, it happened a few times I talked about some thing and an hour later it was talked about on broadcast TV. People over-fit on finding patterns.
https://www.documentcloud.org/documents/24224884-how-voice-d...
https://www.techdirt.com/2024/08/29/cox-caught-again-braggin...
Thank god during the Cold War we had real tech companies, like Digital and HP, staffed by real engineers whose heads were screwed on straight and whose values weren't perverted.
So, there is no general answer to the problem, but you can always approximate it by asking: are you working for the US defense/intelligence industry or one of its contractors? If so, you are almost certainly building something harmful.
Hamilton made a much better case than I could for a way to view the military as a necessary evil neither to be loved nor feared in Federalist no. 8.
1. Is the work that I am doing advancing my personal goals?
2. Is the way I spend my time good for my mental health, my physical health, and my family?
3. Is the work that I do aligned with my ethical worldview?
4. Am I being renumerated appropriately aligned to the value that I am bringing?
5. Am I still learning something each day?
6. Are my family happy with the outcome of this year and the way I contributed to it through my work and otherwise?
If I get a no to any one of these questions, I start looking for a new job. I've moved across country and across the world, and I've been at companies for a long time and a short time throughout my career (that's nearing 20 years at this point). I've switched industries and areas of technical focus many times, and I've even switched entire career paths by simply asking myself these questions after a relatively short period of introspection.
I encourage everyone to consider the ethics of their work, but also to consider regularly whether or not your work is advancing your goals, making you happy, making you healthy, and supporting your family in the way they need to be happy and healthy. Ethics is a core part of this. If you are doing work you don't agree with for purposes you would not agree with, it can be traumatic to your own psyche and deeply affect both your health and happiness, including how you treat other people and especially those closest to you.
Take some time, the holidays are coming.
The author recounts their experience as an intern building software for the Department of Defense, unaware of its intended use. The software, designed to locate WiFi signals, utilized algorithms like R^2, Gaussian estimation, and Kalman Filters to improve accuracy and tracking capabilities. Despite the technical intrigue, the author acknowledges the software’s purpose was to aid in killing people, highlighting the ethical implications of their work.
The author recounts a past experience working on a project for a Department of Defense contractor, where they were tasked with creating a tool to locate phones, ultimately realizing it was intended for targeting and potentially killing individuals. This experience, along with examples like a deceptive quiz and Uber’s “greyball” feature, highlights the potential for code to be used unethically or even illegally. The author emphasizes the importance of developers considering the ethical implications of their work, as software increasingly impacts various aspects of society.
Developers have a responsibility to consider the potential misuse of their work, as unethical applications can have serious consequences. While there is no universal solution, developers should critically evaluate project requests, consider worst-case scenarios, and prioritize ethical considerations over deadlines. Ultimately, developers must decide whether to build a product, even if it has the potential for misuse, or to prioritize ethical concerns and potentially face consequences.
Code can have serious consequences, even death.
Mitch: You did?
Chris: Yeah, and he used to be the number one stud around here in the 70’s. (whispers) Smarter than you and me put together.
Mitch: So what happened? Did he crack?
Chris: Yes, Mitch. He cracked, severely.
Mitch: Why?
Chris: He loved his work.
Mitch: Well what’s wrong with that.
Chris: There’s nothing wrong with that, but that’s all he did. He loved solving problems, he loved coming up with the answers. But, he thought that the answers were the answer for everything. Wrong. All Science no Philosophy. So then one day someone tells him that the stuff he’s making was killing people.
Mitch: So what’s your point? Are you saying I’m going to end up in a steam tunnel?
Chris: Yeah, I am.
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
You don't plan what you don't control, you don't control what you don't own. For the vast majority (yes, including engineers), the only thing we own that is a part of the whole process of capitalist production is our ability to do some kind of work. When that ability is sold (for a wage/salary) to whoever possesses the resources to decide the why, we forego the right to decide how it's used. Just like when you sell someone a chair for example, you don't get to tell them where and how they can use that chair.
You can refuse to get or do this or that job, but you need to have a job. At the end of the day, we all need food and shelter to stay alive and the bills aren't gonna pay themselves. So ultimately, someone—among the vast majority of us who don't have a choice—will also have to do the job you refuse. So we shouldn't delude ourselves into thinking we have some sort of "superpower". We don't. That is, not unless we can collectively withold our ability to work and force the hand of the minority that actually decides why things are done.
A society where those who do the work decide why and how it's done, is a society where the working people own the resources. That is communism. That world is possible, but it won't build itself, you have to tirelessly fight for it. Look over at marxist dot com if you want to help out [1].
That doesn't mean that in the meantime we shouldn't strive to find meaningful work that fit with our values. We should, and whoever manages to find that good for you! You've been lucky (for now) but you won't be forever, and even if you are to be, the vast majority of people with pretty much the same skills and conditions as you, actually won't. We actually have to transform society if we want to secure that for ourselves and for everyone else.
People want more value for more work. If I work 12 hours picking potatoes and have to give 3/4 of them "according to my need" I will refuse to work. If you create fake jobs like "bolt counter" to insure 100% employment so the means can justify the ends (as communism believes) you end up with a collapsing economic system.
The current iteration of late-stage capitalism is not capitalism _at large_. It's feudalism clever disguised as an egalitarian economic system. The solution to this is not communism, it's undoing 100 years of stock-market oriented business design. We don't know what "late-stage" communism looks like simply because a revolution occurs several hundred years before it reaches that point. Not even the premiere implementers of communism, the Russians, could keep their system.
It doesn't work. In any country, in any system, with any group of people larger than 4.
Like you note thoigh, we can have a free market and democracy without that dynamic
Which is why the powers that be have to pass things like this: https://www.govtrack.us/congress/bills/118/hr5349/text/eh to keep people confused, and to conflate being opposed to shareholders (capitalism) with support for authoritarianism etc.
Don't fall for the trap of accepting "late-stage capitalism" as a valid concept - it's a made-up term that has no concrete or meaningful definition and is used by Marxists to constantly move goalposts and impute bad things into our functional (if suboptimal) economic system.