Comment by gonehome | Hacker News Reader

Comment by gonehome | Hacker News Reader

Parent Full thread

I think this is objectively wrong even on a human level. I could see some part of it if you're using empathy in a general enough sense to only mean modeling other minds without caring about their goals except as a way to pursue your own (which doesn't sound like what you're saying). It sounds more like you're putting intelligence in some reference class where you're just stating it's not intelligence until it's already aligned with humans (which is not helpful).

For some reason people tend to think that general intelligence would generate all these other positive human-like qualities, but a lot of those are not super well aligned even in humans and they are tied to our multi-billion year evolutionary history selecting for certain things.

This is basically the orthogonality thesis which I found pretty compelling: https://www.lesswrong.com/tag/orthogonality-thesis - the AGI crowd has a lot of really good writing on this stuff and they've thought a lot about it. If it's something you're curious about it's worth reading the current stuff.

Some other relevant essays:

https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...

https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe...

This talk is also a decent introduction: https://www.youtube.com/watch?v=EUjc1WuyPT8

When we get intelligent machines they will become beings. Contrary to SV corporate-speak, you don’t align beings. You don’t align your neighbors, you don’t align people from other countries. You can’t hardwire them to follow certain principles, that’s the whole point of beings, otherwise you’re back at automated machines. All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?' and that is not a very interesting question to me.

You do align people all the time - what is culture, story telling, arguing over what's good and bad, law? Discussion and changing people's minds? Persuasion and sharing ideas?

> "All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?'"

They explicitly don't ask that because if you get to that point and you don't have them aligned with human goals, you're fucked. The purpose is to understand how to align an AGI's goals to humanity before they reach that level.

Then instead of superheroes think about countries or groups. How do you align Iran? How do you align the NRA? How’s culture, storytelling, arguing, etc. working for us so far? The point is that there is no simple recipe for "alignment", it will be continuous work and discourse, just as it is today with humanity. We’re talking about minds here, not machines that follow exact orders. How do you change minds?

> How’s culture, storytelling, arguing, etc. working for us so far?

Pretty well really (despite many obvious problems) - humanity has done and is doing great things. We also have a huge advantage though that the alignment has a shared evolutionary history so it really doesn't vary that much (most humans have a shared intuition on most things because of that, we also perceive the world very similarly). For the specific examples of countries, international incentives via trade have done a lot and things are a lot better than they have been historically.

> We’re talking about minds here, not machines that follow exact orders. How do you change minds?

We agree more than you probably think? You can change minds though and you can teach people to think critically and try to take an empirical approach to learning things in addition to built in intuition (which can be helpful, but is often flawed). Similarly there are probably ways to train artificial minds that lead to positive results.

> The point is that there is no simple recipe for "alignment"

I agree - I doubt it's simple (seems clear that it is definitely not simple), but like there are strategies to teach people how to think better, there are probably strategies to build an AGI such that it's aligned with human interests (at least that's the hope). If alignment is impossible then whatever initial conditions set an AGI's goal could lead to pretty bad outcomes - not by malevolence, but just by chance: https://www.lesswrong.com/tag/paperclip-maximizer

>Then instead of superheroes think about countries or groups. How do you align Iran?

One could start by not toppling their legimate democratic leader in the 50s, not imposing a dictatorship afterwards, and not sponsoring their neighbors to go to war with them.

Also avoiding subsequent decades of sanctions, insults, condescention, using their neighbors against them, and direct attacks and threats towards them would go a long way towards "aligning" them...

Finally, respecting their culture and sovereignity, and doing business with them, would really take this alignment to the next level...

Reply on news.ycombinator.com