Should computer science researchers disclose negative social consequences?

I would much rather sit in a rocket that has blown up a 100 times on the launch pad but launched succesfully a thousand times more with each time being an opportunity to improve it, than one that has been engineered with waterfall and never flown...

https://spacenews.com/pentagon-advisory-panel-dod-could-take...

These considerations should be taken into account when technology is used, sometimes by policy makers in government.

There's no way industry will police itself, they've proven time and again that profit is far more important that humans.

They are experts in how the technology works. That doesn't make them an expert in how other humans will take advantage of what they have discovered/created. I've taken the charity of assuming the scientist or engineer themselves had non-evil aims.

> This is an interview in Nature. Academics with the ambition to publish in academic journals, be they prestigious like Nature or Science, or smaller and more niche like <insert your favourite journal here>, are taking on a role of a philosopher.

Yes, but a philosopher on which topic? A doctorate from an engineering background is not equally as educated as someone with degrees in history or other studies of human nature in regards to the societal effects of technology.

At least in computing, this seems to have been largely false. Whatever my respect for the expertise of creators, their track record on predicting the future uses of their developments has been genuinely bad. I don't think that's an indictment of their skills, just an acknowledgement of how complex and rapidly-changing systems really work.

Understanding the future impact of ARPANET was impossible without also anticipating the rise of home computers, but the creators of ARPANET had no special expertise on that point. Computability experts and chip designers got predictions about topics from weather modeling to AI Go play totally wrong because everything from GPU computation to neural network advances have rendered their accurate domain knowledge misleading. And the track records of AI experts predicting the real-world role of AI is so infamous that I don't think I need to elaborate.

Honestly, I expect that this would be computer science's Modernist moment. Corbusier and friends were genuinely excellent architects, but the move from "designing on sound principles" to "trusting that we know all the future uses of buildings and designing for those" was disastrous. The result is a legacy of inflexible building design, hostile urban planning, and government-embraced block housing that caused real harm to the people it was inflicted on.

"A better grasp on their topic than everyone else" is presumably true, but it's not enough. Complex systems are often beyond what any one person can predict, and that only becomes more true when they're also rapidly changing from many directions. I don't think most creators in computer science can make predictions good enough to act on until their creations have been exposed to the world for a while.

The small set of engineers with philosophy degrees does not negate the parent's statement.

Similarly tech shouldn't have to consider whether someone with unrelated goals will use their tech for unwanted aims.

Funny how you believe that. Remember the Miami bridge collapse ?

Workers on the ground misplaced one of the supports, and then didn't stop work when loud noises started coming out of the concrete. This was not documented wrong on the design, it was those same companies first mismeasuring the location of the supports, then simply moving them around when they didn't fit.

Who's culpable ? The designers, of course.

Or take Dieselgate. European, German and French car manufacturers systematically and purposefully installed software in the cars to commit fraud on the testers (the government testers knew that this was happening).

Who's culpable ? A single engineer, of course.

Not the regulators that knew what was happening.

Not the European investment bank, who had seen reports on this.

Not Volkswagen management, that had built several facilities explicitly for committing this fraud.

Let's stop pretending the problem with engineering will be solved by assigning blame to engineers. It won't. It can't.

Large scale fraud, committed by management and government is the problem. It cannot be fixed with engineering.

"In my first ever AI class, we learned about how a system had been developed to automate something that had previously been a person’s job. Everyone said, “Isn’t this amazing?” — but I was concerned about who did these jobs. It stuck with me that no one else’s ears perked up at the significant downside to this very cool invention."

"Then, the gatekeepers are the press, and it’s up to them to ask what the negative impacts of the technology are."

Building an entire society of victims and allowing the hysterical press, who are in the business of feeding a hysterical public sounds like a formula for paralysis.

Automotive engineers willfully destroyed opportunity for stablehands, whip manufacturers, and train conductors out of hundreds of other occupations annihilated.

Civil engineers with their oriented strand board trusses, reinforced concrete, and dimensional lumber stole the livelihoods of masons, carpenters, foresters and so many more.

Engineers have centuries of unethical practice to atone for. They need to stop their work because everything that they do is harmful and unethical!

I absolutely disagree. That is why I specifically mentioned culpability for first order effects. If you build a gun for someone you know to be violent you have culpability for its use. What I disagree with is the broadening of this "liability" to include second order effects. That we are now supposed to predict chaotic systems and know in advance all perverse outcomes - and that we should have moral liability should we fail in this task.

Tech was never the issue here - people were. The drive to scapegoat the upstarts should be regarded with great suspicion. While there may be questionable ethics involved with companies it looks to be a classic attempt to scapegoat the troublesome for matters that are peripheral at best and any action they could have taken would have dissatisfied people like how both the left and right are convinced Facebook is biased against them because they don't reflect the world as they wish to see it.

"For instance, machine learning is now widely used for surveillance and advertising in ways that violate basic presumptions about privacy rights."

Those same mathematical techniques are also being used to more accurately detect cancer in MRI scans. Saying everyone needs to be held responsible for second order effects is unrealistic and ultimately damaging to society. Blaming the toolmaker for things they cannot control will only cause no tools to be made.

To me it points to a larger issue in our society that we are on a trend of completely removing context. Context no longer matters, what matters are the effects, regardless of whether you can reasonably be held accountable for them or not. It is thought-crime induced by those who in the same breath rail against that totalitarian behavior and then skip to implementing a version of it themselves.

A number of people on this thread immediately jump to the argument on surveillance and data privacy which is a useful discussion but tangential to what is being proposed in the original article. The article is moral hand-wringing with zero clarity or practicality. And the judge of who is culpable or not is left to a hysterical press, driven by a hysterical and sometimes uninformed public.

Look at what this behavior has done to political (and personal) discourse. Now imagine it being applied to science. No thank you.

> programmers should not try to determine how society will react to technology

That is exactly one of the most important parts of our job.

Do you think doctors should not think about the wider impact of treatment decisions? Legislators should not think about the second-order effects of the laws they pass?

Yes, prediction is difficult (especially about the future, as Yogi Berra said). We all know this. Most of the things that most programmers are doing have known and predictable effects, because most of what we are doing is not new. If we had no idea how society would react to technology, why would we even develop it? Of course thinking about the effect that your work has on the world is a basic responsibility of any human being, how it could possibly be otherwise?

I agree completely that it's hard or impossible to just predict in general how society will react to technology. But in many cases, there are people designing systems specifically for the purpose of getting people to behave a certain way. This implicitly acknowledges that they believe that they can affect behavior in a predictable way. If they can make predictions one way and be correct enough to profit from it, they should be capable of making predictions about things that could go wrong.

I agree completely that predicting something like what Facebook will look like 10 years in the future is impossible, and there's surely a lot of research where the possible consequences are infinite. A lot of results in CS research are like someone developing a new material - "Like, sure, this could be used to make a new, extra deadly tank or something, but it could be used for literally anything." In that sense, it's not useful to make predictions, but I think there's also a lot of research going on at Facebook and Google that looks closer to what I mentioned above, where they absolutely have an intention to change people's behavior and so probably also have the ability to predict the consequences of their complete success.

That this phenomenon came about through the heretofore unprecedented (sarcasm) exploitation of human credulity and vanity, that dopamine spikes are the very mechanism of addiction, and that we react more strongly to things that trip our amygdalae than our cortices, should surprise precisely no-one — least of all the people who were building tools predicated upon those things.

None of this stuff is new. None of this stuff is news. You don't get to play ignorant of the implications of the things you're building. Not while you're selling your time on the basis of how smart you are, anyway.

If we consider the goal of research to improve society, then yes, I think it's reasonable to expect a researcher to spend some time considering whether what they're doing is going to actually improve society. In biology, most funding agencies require you to take a brief (~8-10 hours of class time) course on the responsible conduct of research, even if you aren't doing any human subjects research.

In the specific case you gave of the non-quantum recommendation algorithm, the negative impacts should be more or less the same as the impacts of the quantum algorithm, but with less of a barrier to entry. And even the person proposing this acknowledges: "We all agree it’s hard and that we’re going to miss tonnes of them, but even if we catch just 1% or 5%, it’s worth it."

Absolutely. We shouldn't expect the authors of one paper to decide on the utility of various consequences and alternatives. Also, those consequences don't exist in a vacuum, and depend on other assumptions. Eg: Automating jobs is possibly an ethical positive provided people don't need jobs to survive. And this provides an opportunity to kick off other lines of research and inquiry.

> ...excessive costs from false positives...

Could someone give concrete examples of the possible costs of "false positives"? In the picture I have in mind, papers are not accepted/rejected based on whether someone likes the consequences or not. (as stated in the article)

> It's just possible that researchers should not assume that they're equipped to make long-term predictions or shape the efforts of their peers around said predictions.

I think excuses based on insufficient expertise is a cop-out. There is no social science specialization with predictive models such things. The authors are expected to have the most insight into the specific aspects they are writing on, and generally educated well enough to be a functioning citizen of modern society. If they can vote, they can damn well put some thought into possible consequences.

--

The way I see it, there are two goals underlying this suggestion:

1. Get a better understanding of the possible consequences of different lines of advancement, and, as a community, encourage the "positive" ones while discouraging the "negative" ones. (for some value of positive/negative, which might well be emergent only after much back-and-forth)

2. Get each and every researcher to actively think about the ethics of their research and the problems they choose to solve.

Most commenters are focusing on only the first one; I think the second is equally important. If the focus was mainly on the first problem, we might choose to have editors writing editorials every few months on recent advancements in some topic. Or a conference panel discussing the same. Those will help towards the first goal, but only partially towards the second goal, since most researchers could "outsource" ethics to others.

I would posit that there was no good way to reasonably forecast it at that time. This isn't an irrelevant point, though it will seem that way at first blush. The point is that it's difficult to forecast significantly in advance the consequences of technological advancement. That the people asked to do the forecasting are not trained in analyzing social trends does not make this task easier.

The people behind http://interactions.acm.org/ would say YES. I bet most of academia would say yes.

This is a very long and especially old discussion, I'd say it approaches 150-200 years, and it first started out as a reaction to the optimistic views of positivists like Auguste Comte. Afaik nothing conclusive has come out of it yet, even though the discussion focused on some heated topics during all this time: should the physicists that invented the atom bomb be held responsible for Hiroshima? should the chemists be held responsible for the mustard gas used in WW1? should the bureaucrats with their focus on modernizing the State and having everything well-labeled and well-counted for be held responsible for the fact that that made the Nazis' job a lot easier when they needed to find the Jews among the general population? etc etc

This pigeon-holing was way beyond tiresome, not to mention grossly inaccurate, 20 years ago: please STOP IT. Computer programmers aren't in general some special class of person incapable of learning new skills and behaviours. Quite the opposite for the most part.

I think if we start to delve into the actual implementation of this, and look at real examples, it will be clear that this idea of "research that can be used for bad should be not published" leads to a bad type of society.

What the article is proposing is that creators of new technologies inform the public so that the general public can decide if new laws are required to prevent that new technology from being misused against the public at large.

There’s precedents here and I’ve written a ton of comments about them if you want to read a bit about why such a proposition is the only ethical option.

EDIT: what I mean is, roughly, markets are how we usually screw ourselves up in the first place.

Someone else brought up the culpability of fission researchers for Hiroshima. That's one example of a major technical endeavor where even with historical hindsight people won't all agree on its ethics.

why? aren't we allowed to judge the wrong? nothing wrong in calling someone arrogant if he's arrogant.

Your comment would be much better without its first and last sentences. The inverse of a shit sandwich!

The key point is that without the public’s input, politicians acting rationally, decided to direct their engineers & scientists (also acting rationally) to develop these weapons in secret, with everyone involved knowing that nukes were “extremely powerful” and “terrifying” and “war-ending” but not fully appreciating the whole MAD aspect that happens when your enemies play development catch up.

Tons of technologies today, while not as much of an instant threat to our civilization as nukes, are loaded with ethical concerns that current politicians and the public at large are just not aware of. Just reference the deep fakes saga for one recent example. There’s currently no laws outlawing the use of deep fake techniques to produce revenge porn but there’s every reason to believe that these techniques will be outlawed for such use in forthcoming legislation now that the public and politicians are aware of the risk. Extrapolate that to any number of innovations. With informed debate we can keep laws ahead of the game and prevent new techniques and technologies from being legally used for unethical purposes.

I have a fair amount of expereince, being a biologist who did gene cloning who now works on large scale machine learning models.

TBH I can see a wide range of surveillance-related technologies enabled by CS that are as impactful (but not as explosively so) as atomic weapons (I think you actually mean ICBMs, since atomic weapons aren't an existential threat, while ICBMs are).

I imagine if all 3D printers were distributed with warning stickers saying that they can be used to print guns and knives that more people would try it than if it was less publicized.

Making it just a bit harder to discover the knowledge to weaponize something could give defensive researchers the time to build effective countermeasures.

Also, I kind think the secrecy of the manhattan project was essential, and after carefully reading multiple histories, I am convinced that Leslie Groves was a genius who did more than a good job managing the project. If you look at how he declassified the project, they did everything right. THey had a historian with access to all the data. And scientists who understood the context. And they worked together to release as much information as they possibly could, and even helped make the case for transitioning control of weapons to civilians.

Nobel predicted that the dynamite will end all wars. Turns out, he was very wrong. On the other hand, plenty of technologies in recent decades have created predictions of impending dystopias, none of which materialized.

This all sounds like trying to blame scientists for not telling us the bad people will do bad things, instead of stopping bad people when they're doing bad things.

Since we can't predict the consequences of our actions (few scientists, if polled, would accurately predict the consequences of their papers), it would be entirely speculative.

Imagine a paper that showed an amazing improvement in machine learning- we could suddenly predict diseases just by looking at pictures of people's retinas, or predict any number of other fairly private things. Shoudl a technical reviewer say "I think this could and will be used to discriminate against people, so it should not be published"? I'd far rather the reviewer just say whether there was an error, have the paper be published, and let the faculty of 1000 or the larger community debate the legality of whether the technology could be restricted.

Note that editors, not peer reviewers, have far more latitude to reject papers due to (their perceived, speculative) societal consequences.

I see where you're coming from but that's quite an oversimplification, don't you think? Software that drives a computer's graphic card, for example, which in turn gets used to run software that enables a physician to explore a 3D MRI, are just two examples of tasks that do not consist of automating manual labour.

(You could probably pay a lot of painters to paint a lot of pictures but hooking them up to the MRI might be a little difficult).

There's a lot of software that achieves new things, rather than old and well-known things really fast and without involving hands or a human brain.

I prefer to avoid discussing ethics on forums like HN, so I won't comment on whether or not working on software that puts people out of work is "the right thing to do" -- suffice to say that, if you wish to avoid doing that, there's plenty of room to do it while still doing your job as a software engineer.

By the way, you can always find a "cost reduction" in anything you do, it's a matter of point of view and horizon.

However, while automation helps in the aggregate, the world isn't lived or experienced in the aggregate but on the individual level. Put Jeff Bezos and 9 people living without shelter or money in a room, and in aggregate their average wealth is $15 billion per person; but that number tells you nothing about the individuals. When the factory closes due to technological advancement, sometimes those individuals can't get another job or one that pays nearly as much.

So we need to both encourage technological advances that increase productivity, but make sure it benefits individuals. I think the basic principle is that capital can move much more quickly than people - e.g., you can pull the money out of a factory in Kokkata or Indianapolis and invest bit in one in Hanoi much more quickly than the workers can move to Hanoi (probably impossible) or find another job anywhere. The answer may be much better unemployment insurance, retraining, and laws slowing economic changes enough that individuals can keep up, to a degree based on the economic impact: e.g., closing a big factory in a small town is higher impact than closing a startup in London.

Reading tarted-up flag-planting arXiv papers can be tough going even for practicioners, to say nothing of the general public. But ultimately the general public will have to vote on propositions, or vote in representatives, to pass sensible (not hysterical) regulations.

For example, after an Uber lidar-equipped vehicle killed a Phoenix jaywalker (a separate part of law that I do not like, but acknowledges physics and shitty drivers), Uber halted testing in most markets. This is sensible based on public reaction, but three major problems compounded to take Elaine Herzberg’s life:

1) the lidar did not function as designed, 2) the backup driver did not function as instructed, and 3) Ms Herzberg did not obey the law.

This is a situation where the facts eventually became clear, and still it is difficult to say that one side or the other overreacted (personally I think Uber did the right thing PR-wise, but I also worry that less testing means slower progress towards safer cars, since human drivers are legendarily shitty and much less safe than conservative self-driving cars with functional lidar).

Now let’s consider more complicated problems, such as replacing medicinal chemists with reinforcement learning systems, or China’s AI approach to “social credits”. Both have potential benefits and huge potential drawbacks, and both are relatively easy to grasp compared to some thornier cases. One is purposely obscured due to a dictatorial regime; one is obscured not by administrative fiat but by copyright law and a very silly tradition in academic and quasi-academic research. Hiding results in paywalled “prestige” journals rather than making it available to the general public is seen as a mark of distinction (and avoiding a $30k-$50k open access charge is just sensible conservation of research funds, in this scenario. I know because I am an academic and I personally think OA charges are a waste, though for different reasons.)

Anyone who wants to [break the law] can obtain the primary documents in the former case via sci-hub. That’s just a fact. But is this any more reasonable than Georgia allowing companies to charge for access to state laws? Is it reasonable for people to pay twice, in most cases, first for the research* and then for the report, when faster dissemination of primary sources can directly inform debate over some important issues?

It seems silly that we would have a system where others are expected to pay for, digest, and opine upon research done by the primary authors. It would not be necessary if the norm was to disseminate AI breakthroughs in a format like distill.pub, where the whole point is to explain both the results and their context. But until academic and quasi-academic researchers can kick the “prestige” habit, that’s unlikely to happen, and the public will be left to listen to pundits (often idiots) rather than forming their own well-informed opinions, and that sucks.

No time to edit, but hopefully you get the idea. Disclaimer: I am an academic, applying “AI” to medical care for children with rare and deadly diseases; I used to work at GOOG a million years ago; and what I see among the worst hucksters sickens me, because I think it’s going to kill people in spite of better competing ideas that could improve the human condition. I’m just one person with one vote. I’d like everyone else to form their own opinions, because theay might change mine. Thus primary sources are critical.

* anyone who does not believe that taxpayers subsidize industrial & corporate r&d via tax credits and incentives is quite naive; it is not different from academic competition in that regard.