Hacker News Reader

Comment by mariusz79 | Hacker News Reader

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us all. But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?

View on HN

You've got this normative halo around "intelligence". All intelligence is, is an agent's ability to satisfy its utility function.

>But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?

That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the existential issues of creating paperclips long after the last piece of stationary was destroyed would be judged of negative utility.

Is Ebola virus intelligent? It has a goal - reproduce - and it's trying to maximize it through any means available to it.I don't think we can say that the virus is intelligent. If it continues to evolve, at some point in time it could potential (highly unlikely but still possible) become intelligent, but will it become self-aware at this point? And if it does will it still want to replicate uncontrollably or will, just like humans, decide the it needs to better manage it's environment?

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc etc etc are all stupid, clearly self destructive behaviors at any level from abstract ethics to species survival. But we come with a bunch of hard wired root goals (like "yum, candy" and "babies are cute" and "seek power") that can't be switched off and are incredibly hard to reason with, and their consequences are already an existential threat. And that's just with dumb, slow-moving humans.

I guess my assumption is that increased intelligence should bring with it increased self-awareness.

If it doesn't, then... well... like I said elsewhere in the thread... put that on your list of potential depressing answers to the Fermi paradox.

Your mistake is assuming that self awareness is sufficient. "To know" is separated from "to choose to act" by preference, and preference is based on goals, and the ultimate, root level goals of a being are not likely to be something it wants to rewrite, even if it has the ability, any more than you'd willingly take a pill to become inhuman.

But we are not "super intelligent" ;) The truth is that this whole discussion does not make much sense if we have different understanding of AI. For me a super intelligent AI is an entity that is able to reason, notice any potential problems and come up with solutions. Paper Clip AI on the other hand is smart enough to know how to create paper clips and create whole manufacturing process of converting humans into paper clips but stupid enough not to understand, or care why it's doing that and not knowing that these paper clips will have no use if there are no humans left.

In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.

Reply on news.ycombinator.com