> The problem is that you've instructed it to maximize Facebook's corporate interests. How do you leave room for "the intentions of its creators" while doing that? There's an inherent contradiction between putting a giant override button on your superintelliegence's goals but not overriding them all the way.
Well, assuming it is created by Facebook, then "maximize Facebook's corporate interests" and "the intentions of its creators" are actually the same thing. There is no contradiction.
"Facebook's corporate interests" is a phrase in the English language which refers to a complex concept. If you've told an SI its objective is to "serve Facebook's corporate interests", in order to fulfill that objective it needs to understand what the phrase "serve Facebook's corporate interests" actually means–which implies understanding a lot about human society, what kind of entities corporations are, what "corporate interests" means (the interests of management? the interests of shareholders? etc), what the interests of human beings are (since management are human beings, and shareholders are directly or indirectly human beings too), etc. If it actually is an SI, it should have no trouble comprehending the full breadth of what the instructor actually meant by the instruction, as opposed to implementing some overly literalistic reading of it. Humans give each other orders all the time (the workplace, the military, government bureaucracies, etc), and humans are most of the time pretty good at understanding the intention behind orders and implementing the intention rather than reading it overly literally in such a way that actually undermines that intention. Yet you posit the existence of an SI then assume it will do a worse job than humans do at correctly following orders, which contradicts the idea that it is an SI.
> Sure. How certain are you?
How certain can I be of anything? Maybe I am wrong and an anti-human SI will destroy humanity. Maybe tomorrow I will die due to a heart attack or stroke or fatal car accident. The latter is far more likely than the former, and there's more concrete mitigation strategies available to me too.
In terms of existential risks to humanity, I think asteroid impacts are a far more concrete risk than anti-human SIs, so if we are going to expend resources in mitigating existential risks, the former is a better focus of our efforts. We know extinction-level asteroid impacts have happened before and sooner or later will happen again. Anti-human SIs are such a speculative concern, we can't even be particularly confident in the correctness of our own probability judgements with respect to them; nor can we have much confidence in our judgements of how effective proposed mitigation strategies actually will be. I think we can have much more confidence in our ability to develop, evaluate and deploy asteroid defence technologies, if the world's governments decided to spend money on that.