Hacker News Reader

Comment by pointlessone | Hacker News Reader

This is actually one of the failure modes described long ago. Paperclip maximizer. An AI that has agency, intelligence, and a goal. In the course of achieving its goal it completely neglects all other considerations. These unaccounted externalities are the danger.

View on HN

In addition to a hypothetical AI, it sounds like you are describing real companies here.

And likewise, corporations have already been described as AIs: https://patternsofmeaning.com/2017/11/30/ai-has-already-take... et al.

You're not wrong but also real companies consist of people. Individually they share many of their thinking patterns and moral views with people outside of those companies. So even though collective nature of companies diffuse responsibility and somewhat erode moral considerations of collective action it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. AI doesn't share thinking patterns. We're not even sure to what extent it can do that. We almost certainly don't operate on matrix multiplication while most AIs do. We don't know if AI can share our moral considerations. Many people act as if they equate intelligence with morality, as if one can not exist without other, and as if there's only human kinds of morality. Meanwhile ChatGPT will lie to your face without a hitch even though OpenAI spent a lot of time trying to make it not.

>Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth.

This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.

It is not a counterexample. Most individuals are having hard time taking into account consequences that are decades removed and either vaguely defined (e.g. 3℃ global temperature raise is a catastrophe) or too great in scope and scale (e.g. literally global societal collapse). But even so externalities are catching up. British Petroleum is pivoting into solar, for instance even if for PR reasons. As a result of external pressure (public opinion and legislation/regulation) fossil fuel industry is in decline. Most major companies are trying to diversify and do something else instead. Companies that comprise fossil fuel industry will probably remain but they'll do something else.

You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.

Whether or not BP works as a counter example has no bearing on other examples. Aramco and ExxonMobil certainly look like a counterexamples to me.

Adding on to this, companies as “AI-orgs-running-via-human-individuals” can be checked by other similar entities of comparable competency, e.g. activist groups, governments, non-profits, and so on.

> it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth.

Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.

Reply on news.ycombinator.com