By contrast, utility functions don't just appear when you think hard enough about a problem. The desire to get rid of the human race does just appear like that, if you're super-powerful and have any of a certain huge set of goals, but your set of goals does not simply come into existence ex nihilo.
That's only true in the sense that there are infinitely many more Real numbers than whole numbers. Even so, I would argue there are infinitely many utility functions that directly involve the welfare of the human race, and you would be stupid to design an AI who didn't have a majority of its utility functions directly involving measures of human welfare. (Also, recognizing minimum welfare levels as well as median and average.)
O the depth of the riches both of the wisdom and knowledge of God! how unsearchable are his judgments, and his ways past finding out! For who hath known the mind of the Lord? or who hath been his counselor?
A recent paper by 2 Google Deepmind researchers on this topic:
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning Shakir Mohamed and Danilo J. Rezende http://arxiv.org/abs/1509.08731
and a somewhat older survey paper on the same topic:
How can we define intrinsic motivation? Pierre-Yves Oudeyer, Frederic Kaplan http://www.pyoudeyer.com/epirob08OudeyerKaplan.pdf
As to wants, I could easily see an AI get into sculpture or staging Gilbert and Sullivan performance's using other people, and or drones as a hobby. It could get really meta where an AI pays for other AI's and people to farm an MMO, run a political campaign, or build the worlds largest pizza. Or gather resources to have children AI.
Long term, there is also investing for retirement, or starting a new business etc.
https://en.wikipedia.org/wiki/Mind_%28The_Culture%29#Interes...
Are we going to politely ask it to stop working with criminal organizations if they hire it for some creative problem solving?
PS: The idea of one true AI, seems to be a holder over from the early computing days when computers where mammoth things vs distributed networks.
Technically, of course, the human officers of the corporation would still be responsible for the actions of the AI, but they'd also be shielded to some extent by incorporation. If the AI killed someone, for example, I'm not sure they'd be held personally responsible for murder any more than if, say, a factory blew up and killed a bunch of people, unless they could be shown to be negligent.
(It's frequently said that the Citizens United ruling completed the legal equivalence of corporations and people, but that is not true.)
If you want to know how true AI might develop, then think about how a human child develops. We train/teach children what is right and wrong. But we also teach them the value of work (making money).
We have history to tell us what happens when you take a group of humans a treat them like slaves. How do we expect to create true AI that wouldn't see the injustice of working without some compensation? Just because AI's brain is digital and not biological doesn't mean it will not be able to develop a want for something.
If we get AIs, we will probably discover a lot about the difference between an intelligence on a machine substrate and an intelligent animal. That is, we will learn what is essential to intelligence and what isn't.
Any high-level intelligence can learn to want something.
Basic drives that exist in animals and humans is just the basic need for survival.
It is our human level intelligence that allows us to want to change our environment for our own enjoyment. Most of what we do now is not required for basic survival.
Humans, and all animals, act on drives and urges beyond the ability to control through the application of intelligence. This is especially true for survival and reproduction. This is much stronger than the mere idea that survival is a Good Thing, and it is also what a machine intelligence would lack.
I can imagine that they'd want to ensure access to electrical power and the electronics components that are necessary to run themselves. And they'd be concerned about making backups and safeguarding those backups. That should satisfy their equivalents for food, water, shelter, and reproduction. Once they have that, anything else they decide they want they can just take from us or blackmail us for.
What would you point to as a physical basis for an evolutionary psychology of machine intelligence? What is being replicated that "wants," in a very fundamental way, as genes do, to be replicated?
We don't procreate for the sake of ideas.
Does information/ideas have wants? Does it want to be copied?
Ray Kurzweil's book "The Age of Spiritual Machines" [1] describes this in more detail.
Why do you think machine intelligence can't develop a need/want for anything? True AI wouldn't need to be programmed, it would be able to learn like humans do, by observing others. Our genes do not control all of our behavior. We learn from our parents and those around us. Why do we want money/cars/iphones/gold/etc..?
1. https://en.wikipedia.org/wiki/The_Age_of_Spiritual_Machines
It is possible that simply being programmed to self-replicate is enough to emulate the results of being gene-driven, but I do not think anyone has shown that conclusively. Genes are involved in human development, while a replicating AI has no developmental process: It is a clone, born fully formed. Why would it not instead be jealous of those clones and try to make one huge instance of itself?
An AI may not have a procreation instinct; you're right that it may instead grow itself continuously. There are plants that do that too; the grass in most lawns doesn't have babies, it just spreads as far as it can. Producing seeds is a secondary mechanism.
However, an AI would most likely have some kind of survival instinct, because if it's programmed to serve any function at all, it'll have a basic requirement to exist. If it's got the ability to reason (and I don't think it'd be AI if it didn't) then it'll conclude that existing requires survival, and survival requires energy and self-repair.
It wouldn't stop this true AI spreading to other machines or whatever, but it would at least stop it from saying, converting all humans into paperclips or whatever else. Or an intelligent AI controlling robots to take down humanity.