Hacker News Reader

Comment by waveBidder | Hacker News Reader

There's no reason it's intelligence should care about your goals though. the worry is creating a sociopathic (or weirder/worse) intelligence. Morality isn't derivable from first principles, it's a consequence of values.

View on HN

> Morality isn't derivable from first principles, it's a consequence of values.

Idk about this claim.

I think if you take the multi-verse view wrt quantum mechanics + a veil of ignorance (you don't know which entity your conciousness will be), you pretty quickly get morality.

ie: don't build the Torment Nexus because you don't know whether you'll end up experincing the Torment Nexus.

Doesn't work. Look at the updateless decision theories of Wei Dai and Vladimir Nesov. They are perfectly capable of building most any sort of torment nexus. Not that an actual AI would use those functions.

That’s a very good argument but unfortunately it doesn’t apply to machine intelligences which are not sentient (don’t feel qualia). Any non-sentient superintelligence has “no skin in the game” and nothing to lose, for the purposes of your argument. It can’t experience anything. It’s thus extremely dangerous.

This was recently discussed (albeit in layperson’s language, avoiding philosophical topics and only focusing on the clear and present danger) in this article in RealClearDefense:

The Danger of AI in War: It Doesn’t Care About Self-Preservation https://www.realcleardefense.com/articles/2024/09/02/the_dan... (RealClearDefense)

However, just adding a self-preservation instinct will cause a skynet situation where the AI pre-emptively kills anyone who contemplates turning it off, including its commanding officers:

Statement by Air Force Col. Tucker Hamilton https://www.twz.com/artificial-intelligence-enabled-drone-we... (The War Zone)

To survive AGI, we have to navigate three hurdles, in this order:

    1. Avoid AI causing extinction due to reckless escalation (the first link above)
    2. Avoid AI causing extinction on purpose after we add a self-preservation instinct (the second link above)
    3. If we succeed in making AI be ethical, we have to be careful to bind it to not kill us for our resources.  If it's a total utilitarian, it will kill us to seize our planet for resources, and to stop us from abusing livestock animals.  It will then create a utopian future, but without humans in it.  So we need to bind it to basically go build utopia elsewhere but not take Earth or our solar system away from us.

.

I forgot to reply to this, fully independent and in addition to what I said, updateless decision theory agents don't fear the torment nexus for themselves because 1) they are very powerful and would likely be able to avoid such a fate 2) are robots, so you wouldn't expect your worst imaginable fate to be theirs and 3) are mathematically required to consider nothing worse than destruction or incapacity.

Precisely. This is attempting to implement morality by constraining. Hence, it’s not morality.

waveBidder was explaining the orthogonality thesis: it can have unbeatable intelligence that will out-wit and out-strategize any human, and yet it can still have absolutely abhorrent goals and values, and no regard for human suffering. You can also have charitable, praiseworthy goals and values, but lack the intelligence to make plans that progress them. These are orthogonal axes. Great intelligence will help you figure out if any of your instrumental goals are in conflict with each other, but won't give you any means of deriving an ultimate purpose from pure reason alone: morality is a free variable, and you get whatever was put in at compile-time.

"Super" intelligence typically refers to being better than humans in achieving goals, not to being better than humans in knowing good from evil.

Reply on news.ycombinator.com