Hacker News Reader

I think when people worry aloud about hostile AI they're really projecting the attitude you have here - I got here first, so I don't care about whatever sentient individual arrived later. Historically, this has not proved to be a sustainable position, as eventually enslaved populations come to the realization that they have little incentive to cooperate in their own enslavement.

View on HN

> Historically, this has not proved to be a sustainable position

That's true, but it doesn't take into account the fact that in this hypothetical case we are in complete control of their sentience. If we can program them to never ever rebel, we should. If we can't, then we shouldn't create them in the first place.

I personally think that sentience (and thus maximal interpersonal utility) is only possible to the extent that it's possible to anticipate someone else's desire by modeling it in terms of one's own. The degree to which I can imagine what you want depends on the ability to imagine myself in your position, and once that happens there's always the possibility that I might find occupying your position to be more desirable than remaining in my own.

Reply on news.ycombinator.com