I think you are homing in on a key philosophical difference between me and LessWrongsters. Don't really have time to get into it now, except to say that it is kind of arrogant to think you can design or think about superintelligences without reference to the best existing intelligent systems we have. Especially if you want to keep them goal-compatible. The pathologies of such systems are especially instructive.