This is of course all about goal setting. I think those fearful of maximisers are being rational, but if we’re going to build AIs then we need to make them part of our community. The things that stop us as individuals going off the rails and taking everyone else around us down with us is respect for community, for others rights and interests. The need to seek consensus and consent. Figuring out how to implement that is the real challenge.