for some reason that one wasn't even included in the list of books in the series on the inside jacket of the other books that I had.
I remember I had to really hunt for it and it was from a different publisher. never knew why.
I don’t think that’s the main problem, there are a lot of moral dilemmas where even humans can’t agree what’s right.
If each human could pause the state of the world and gather all information and then decide, they would act humanely
Just think of abortion, wars or legalizing drugs. People disagree completely over those because nobody agrees which choice would be the moral one.
HPMOR offers a solution called 'coherent extrapolated volition' – ordering the super intelligent machine to not obey the stated rules to the letter, but to act in the spirit of the rules instead. Figure out what the authors of the rules would have wished for, even though they failed to put it in writing.
We are debating scifi, of course.
What if the original author was from long ago and doesn't share modern sensibilities? Of course you can compensate when formulating them to some extent, but I imagine there will always be potential issues.