My personal feeling is that nobody has any idea what they're talking about. And when I say nobody, I mean NOBODY including Ng, Musk, et al.
The problem is with those who read others and just assume they're "experts" or their opinions have value. In some areas they do, sure. In this area they most likely don't.
That, or those of us who are skeptical are uninformed.
Or it's somewhere in the middle.
In any case, I'll stick to my skepticism for one main reason, which is, general intelligence is supposedly modelled after human intelligence. And human intelligence is something we're JUST BEGINNING to scratch the surface of understanding while at the same time, really have ZERO idea how deep the rabbit hole goes.
And as any competent engineer here should know, when trying to emulate a natural model, you first need to understand the model. Until we do that, we will not create a general AI. Like any other computer program (WHICH IT WILL BE! in this conception), it needs to be programmed. We need to know what to program before we can do it! Computer programs don't emerge spontaneously.
Is it a given an AI would share them?
I think we're not really talking about AI at all - we're talking about our current economic and political systems, which appear to have many of the properties we're imputing to evil AIs, but for some reason are far less criticised and debated than hypothetical machine monsters.
Basically we're screwed if it's trying to maximize anything that depends on physical resources. We're also screwed if, e.g. it's trying to maximize human happiness, and achieves it by lobotomizing us all into happy idiots. There are all sorts of ways we could screw up AI motivations, to our own detriment.
That assumes there's only one AI, whose crazy motivations will be unopposed. But if there are multiple AIs, it's even worse; they will compete and evolve, and the only ones that survive will be the ones that do maximize their resources, and jettison any niceties about preserving human life.
If an AGI can change its own goals, that just means we can't control it at all. There's no reason its goals have to include human survival.
A decent enough AGI with access to resources would probably figure out it is costly to wage war.
Heck, an equal probability is that an AGI will follow a bounded version of zeroth law or even nonviolence.
That is assuming it does not place value on humans based on our history of research and development. I would expect any action more akin to forced upload or upgrade instead. Anyway, at that point we might not resemble present day humans anymore.
Do you think that sufficiently smart GAIs must be non-rational? The change of its goal will inevitably make its original goal less likely to realize. It is not rational.
> should quickly figure out that maximizing anything is the way to ruin - running out of resources.
Are you aware of the concept of maximization of expected utility? When AI will figure out that it can run out of resources, it will reallocate part of the resources to acquire more of them.
How can action, which modifies the goals of the AI, be the result of argmax_a E(a)?
E(a) is expected utility of action a
(Hint: Emax is not, most hill climbing algorithms are not. They both get trapped in local optima.)
Sometimes you need a few good lies (false hypotheses and bad attempts) to actually arrive at the truth.
Agi sort of means, no matter what endeavor, you will be worse than the ai. There is no way to know how we will be treated. Maybe we are useful in some way. Maybe they'll kill us off but simulate our lives so we don't 'really' die. Maybe that's their morality.
Maybe they ignore us. I don't care about ants. I walk around and step on them, but not intentionally. On the other hand, if there are ants in my house, I eradicate them.
Who knows.