Hacker News Reader

Comment by Barrin92 | Hacker News Reader

There's a great article by Ted Chiang about this[1], comparing the psychology of the paperclip AI scenario to the single-minded "increasing shareholder value" mentality of business, in particular, silicon valley startups.

He points out that the companies of the very people who fear the ai-apocalypse do already what they're accusing AI of, pursuing growth and 'eating the world' at all cost only to maximize narrowly defined singular goals.

[1]https://www.buzzfeednews.com/article/tedchiang/the-real-dang...

View on HN

I’d actually turn both this and the original article around, and say that corporations behave like AIs. They are artificial. They are (variably) intelligent.

The functions of processing inputs to create meaningful outputs are carried out by neurons made of meat (human employees) and of silicon. These processes are governed by rules and metaprocesses, which the entity continuously optimises and improves, in order to further its own growth and to improve its fitness for its purpose - typically, enhancing shareholder value. I cannot think of a single criteria for intelligence or even life that a corporation does not possess, from independent action to stimulus response to reproduction to consumption to predation.

I think the first true hard AI will emerge accidentally, and it will be a corporation that has largely optimised humans out of the loop - but even with humans as a component of a system, a gestalt, an artificial supra-intelligence, can still emerge.

This also neatly sidesteps the whole question of “should AIs have rights?”, as corporations are already legally persons.

I agree: http://beza1e1.tuxen.de/companies_are_ai.html

It leads to an interesting question: Is a corporation controlled by its CEO?

Nice - that overlaps pretty much perfectly with my reasoning!

In my view, no. The CEO is at best a race condition breaking function, at worst, a parasitic infection.

If corporations are self-optimising AIs, then they will optimise CEOs out of existence if they aren’t conducive to fitness.

Does that not suggest that CEOs are worth their cost to the vast majority of companies that retain them? What companies operate without a de facto CEO?

Well, CEOs may be inherited from non-optimal initial conditions of early ancestors and simply haven't been replaced by a better competitor yet. Or they have been and we don't understand that yet. Naturally occurring genomes encode for processes that result in all sorts of things that don't actually have to occur. The appendix, the frenulum, most of the DNA itself, as far as we can tell. Nature's full of random dead-ends that keep propagating because the system hasn't gotten over the barrier of the next local minimum yet.

Chicken can run faster if it had optimized its head away. Because it can ignore limiting factors like potential obstacles in its current course. But for how long?

Actual problem solving in a corporation is done by humans. Corporation policy can provide a set of incentives to align goals of its employees with the goal of the corporation and a structure to more or less effectively combine specialists' efforts. But someone had to think about the corporation as a whole and change its policies if the need arises.

Cephalization is a known evolutionary trend after all.

CEOs are controlled by their context, i.e. owners and regulations (for publicly-traded companies) which drive the profit-motive, and then have agency to take various actions to achieve this goal.

I have a strong feeling that if we went on like that and one day there will be an AI which will optimize you out - "Humans do not add much value to things, they are just cost centres."

Humans add all the value and always will, because that's what value is - what humans want. If an AI has some goal independent of humans, it's no different from a car rolling downhill because the parking brake failed.

Sure, but good luck convincing an AI that after it has power. It's important to build the AI to think that way from the start if it is capable of getting power over us.

Reminds me of this great SMBC comic -https://www.smbc-comics.com/comic/2012-04-03

We need to be careful of this kind of utility optimization, even when it readers to humans.

It's been a long time since I took a course in philosophy, and all I recalled was John Stuart Mill being associated with utilitarianism.

Looking at his Wikipedia page, it appears that his idea of the utility associated with pleasure was independent of the specific person. If X is the amount of pleasure humanity gets from something, and Y is the number of people that exist, then the pleasure an individual is assumed to get is X/Y. If I understand correctly.

If someone were to abuse the concept of utility by considering it to differ between people, it seems to me a lot more likely that they would say something like "I get a unit of utility from this candy bar, but many other people do/would not, so I don't need to share with them", rather than "I get millions of units of utility from this candy bar and others only get a normal amount".

Great quote: "There are industry observers talking about the need for AIs to have a sense of ethics, and some have proposed that we ensure that any superintelligent AIs we create be “friendly,” meaning that their goals are aligned with human goals. I find these suggestions ironic given that we as a society have failed to teach corporations a sense of ethics, that we did nothing to ensure that Facebook’s and Amazon’s goals were aligned with the public good. But I shouldn’t be surprised; the question of how to create friendly AI is simply more fun to think about than the problem of industry regulation, just as imagining what you’d do during the zombie apocalypse is more fun than thinking about how to mitigate global warming."

The narrative that the danger is corporations might accidentally let AIs gain power is appealing but indeed ought to be challenged.

The paperclip/strawberry monomania fear masks the way things how things are shaping up now. AI is far from being to able intelligently plan over a time horizon - maintaining a real world process all the way to even a smallish goal requires real world corner-case handling that the best AIs haven't got (self-driving cars have five years away for 10-15 years, etc).

But AIs as "intelligent" filters are here now. These allow "better" decision making along with unaccountable decision making, both of which indeed fit well with the agenda of the modern corporation. And that's the thing - the modern corporation is already short sighted, the process of ignoring the potential for climate happened well before automated decision making appeared. But the modern corporation is still more far seeing than a deep learning network and still theoretically and legally accountable to society. However, indeed, deep learning networks that only maximize those qualities that these corporation want maximize and ignore other considerations suit the indeed short-term preferences of the institutions.

In which a science fiction writer weaves a fantastical tale about how the Doomsayers are making their decisions from fantastical tales and emotions, instead of actually reading and responding to the arguments they present. The irony is so palpable that if it wasn't so disingenuous, it might even be funny.

Here's some actual insight on the topic: https://www.youtube.com/watch?v=L5pUA3LsEaw

The article has some neat points but I feel like it's confusingly tugging in opposite directions. It's weird to assert that AIs couldn't possibly be accidentally made to be singularly-focused on a task that's harmful in excess, but groups of people as corporations totally could. But also AIs and corporations are the same. But also only sociopaths would worry that AIs might possibly be accidentally made without human concerns, that's science fiction, only groups of people could manage to operate without human concerns.

I think the point of thinking about how we regulate corporations so they don't focus on singular goals in a harmful way is important. I just find the article's framing weird, as if it thinks people can only care about only one of AI alignment or corporation alignment, and that it's desperately important that they pick #2 instead of #1. I think the topics are actually very related and it's actually useful to think of them as different examples of the same problem. Thinking about AI alignment can help us understand how corporations should be regulated, and vice-versa.

(I think AI safety/friendliness/alignment problem is important, but it's really only relevant to human-level AGI. It's not about modern-level AI / machine learning / algorithms / whatever. No modern AI/ML understands the world at a human level and generates its own goals and motivation in a general sense. There's plenty of ethical issues about how we use modern AI/ML in businesses and so on but it's pretty separate from the AI alignment problem. The AI alignment problem is only an imminent issue further down the line, like "how do we build moonbases that don't fail?", which is interesting, and I'm tired of tons of articles bringing up the alignment problem like "There's some weirdos wondering about how to make buildings in space, but don't they know there's nothing to build things out of in low-earth orbit?? Are they just delusional?".)

So the corporations project their own immorality on AI.

Only if embedded in the cost function or the specific implementation.

AI has no concept of morality.

Really the Paperclip Maximizer bugs me as an example for all of the wrong "missing the point" reasons like Buridan's Ass in that a donkey wouldn't starve to death between two equidistant hay piles.

Anyone who sets it up to maximize raw output of very simple commodities is doing things incredibly wrong economically in the first place. They would go out of business in the first place paying for storage costs alone - let alone the access of everyone else to counter the rogue self-replicating.

It would be admittedly silly as an objection for the metaphor if not for the fact people really do believe AI would be that way.

>They would go out of business in the first place paying for storage costs alone - let alone the access of everyone else to counter the rogue self-replicating.

These things reduce the total number of paperclips produced, and therefore would be accounted for by such an AI. The Universal Paperclips [https://www.decisionproblem.com/paperclips/index2.html] game has a pretty good take on it.

Reply on news.ycombinator.com