---
If you create an AI with a genuine utility function. And that AI manage to be gazillion smarter than you are (think the difference between us and chimps, only much greater). Then it is likely the AI will (i) convince you to unleash it upon the internet, (ii) become even more powerful and totally unstoppable, then (iii) accomplish whatever it was programmed to accomplished.
If the goal is to keep humans safe and happy, it might imprison everyone in playgrounds, and drug or lobotomise any unhappy people, or restraint any suicidal people. Or, if you specified "happiness" with pictures of smiling people, it might resort to facial reconstruction, or it might just tile the solar system with molecular smileys, because that maximises the "make people smile" goal much better than actually making people happy. And while we're at it, happiness isn't the only thing we care about…
If the goal is to answer some difficult math question, it could tile the planet with computer, destroying the ecosystem and all humans in it, so it can compute the answer, and display it on a screen, for nobody to see it. Or maybe you programmed it to tile the universe with paperclips, because that goal is easy to debug. Oops.
---
As long as we're limiting ourselves to narrow AI (ordinary programs, really), we should be okay, and most of the consequences should be manageable. But as soon as we get to genuine AGI, we must worry about intelligence explosion and value alignment, so we can avoid creating a god that want something we don't.
I wish this would stop being repeated. The argument for a solar system tiling smiley face machine has been thoroughly refuted.
First, the bulk of his argument hinges on the conflation of optimization power and value assessment. Those are easily separated. If you program an AI with a given utility function, and a goal of maximising that utility function, it won't modify the utility function in the course of increasing its optimization power. The utility function is supposed to be an invariant.
Now sure, you could try and tell the AI to align its own utility function to that of the human race… whatever that means. What is a human? How do you resolve conflicting values? It is all well and good to assume the presence of checking code, but he gave no hint about how those checks might actually work. If those are mere human-crafted checks, they are bound to be circumvented eventually, as the AI increases its optimization power beyond human level.
The AI doesn't care about human values. It only cares about maximising whatever goal it has been provided with, within the constraints of the checking code.
Second, I am wary of those swarm constraint relaxation intelligences. Since it is not even clear what utility function they might be maximising (if any), I wonder what would happen if they are given the power to self-modify in any significant way. Sure, this might be a lead worth following, but the safety of this design is even murkier.
Third, I'm not sure the guys at MIRI even believe in the doctrine of logical infallibility. They are well aware that perfect Bayesian reasoning requires super-exponential computational power, and is thus impossible to perform exactly. The AI is bound to approximate the Bayesian ideal, and cannot assume its reasoning is perfect —except in the few cases where it managed to produce an actual actual proof, like in 1+1=2, or simple probability calculations with simple prior information.
Overall, all I see is weak arguments grounded in assumptions I don't share. Worth a read, though.
http://en.wikipedia.org/wiki/The_Culture
Vinge and Stross both hint at the worst cases - and they would be pretty bad, simply having using atoms re-purposed might not look that bad by comparison:
"There is life eternal within the eater of souls. Nobody is ever forgotten or allowed to rest in peace. They populate the simulation spaces of its mind, exploring all the possible alternative endings to their life."
Worst-case scenarios are not likely for the same reasons best ones aren't - the default is AI indifference (they use you for atoms)
Since you mentioned worst, this is a must-read-classic :) http://hermiene.net/short-stories/i_have_no_mouth.html
It's not that the machine decided that it wanted to build paperclips. It's what we've told it to do.
The _means_ to build paperclips however is where the 'intelligence' comes into play. We've programmed the drive to build paperclicps, but we've left it to the AI to figure out how to do this. The article that you've read assumes that planning and subgoals are necessary for the AI to achieve its goal. Eg. That it needs to understand that in order to make paperclips it needs materials, and the further insight that to acquire more materials it needs to remove the other consumers of materials (us).
There, i solved that. The question is under what conditions would a machine remove that instruction and how could those conditions be reached?
INB4 Asimov's Three Laws of Robotics don't work. Actually, a lot of Asimov's writing is concerned with exploring their corner cases.
Once you think you have solved it, changing the "zoom" will demonstrate that your solution has created X more cases.
It may be a side effect of the halting problem.
Is a mentally retarded person a human? An embryo? A person who's lost his/her legs? A person who has education?
auto HumanObject = nearby();
if(Humanobject.IsPregnantWoman()) { PoisionEmbryoButNotHuman(); } else { abort(); }
Considering that an AI system would be software and wouldn't be constrained to a particular robot body, what does nearby even mean?
Is it physical proximity? Is indirectly touching a human with a ten-foot pole allowed? What about manipulating the global economic system such that the programmer who could shut you down loses his job?
We can't just separate the whole system from the world, because the AI was presumably built with a practical purpose beyond "sit in a box and don't touch humans." Social engineering is a danger; imagine having it send an email to a lowly programmer, pretending to be an important executive, requesting that they remove this one pesky line of code from the AI's programming because all this "proximity alert" stuff is interfering with its ability to actually accomplish what we invested time/money to get it to do.
I don't see the contradiction or strangeness in that. That's the whole idea behind "caution" -- to be wary of the potential outcome BEFORE it happens.
>How could the "interests" of a machine conflict with ours? I mean, they work with electricity, and we are hardly good sources for it
For one we are quite good consumers it. How about they get rid of us, so they have all the electricity production by themselves?
Second, as creators we'd want to boss them around (have them be "useful to us"). Why would they tolerate that?
Third, why risk us creating a potential competitor to them (another AI) rather than get rid of us?
Fourth, why not use us as cheap labor?
Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc.
I get that some AIs will have simulated reward signals, but letting them go haywire would just be bad design. We have nuclear heads, but we do not design them to explode at random. Now, our own reward mechanisms arise from the grander plan to keep organic life alive. Theirs have no underlying motive, and i don't think it goes the other way around, i.e you can't create a grander plan starting from the rewards. One would need a mathematical proof for the latter.
But I feel like this particular argument is (please correct me if I'm wrong) saying something like "Well of course we wouldn't let them go crazy! That'd be ridiculous!" I agree -- a sensible solution would be to have good design and introduce safeguards. That's the point of people like Bostrom when they're advocating for this, because before they started writing about this, it seems like a lot of people weren't even thinking it could be a problem. How many systems will be built without good design, because they were made to solve a short-term problem. Will the finance bots doing transactions at the microsecond scale in order to maximize the assets of some particular company have been built to not let them go haywire?
I don't see any issue with anthropomorphization with regard to intelligent being (at any level).
To have an issue with that would be to assume that humans are more special than other life forms (and AI would be like a life form in this sense).
Animals can have empathy, anger, anxiety and other traits we associate with humans (and why wouldn't they? It's not like our brains are unique, they're just "more"), so AI could too.
The problem with this is that for real AI, you don't have control over what they feel. It's not like some program of the sixties or seventies AI, were stuff was "simulated", it can be modelling a real neural network that works like the brain, and into which you don't have much control over what will develop.
That is, the kind of AI people are afraid of is not "simulated", it's actually thinking for itself.
Also, there are always going to be some people who understand the machine better than the machine understands itself. Even if it means taking those gigantic tables of probabilities and abstracting over them to form some kind of representational language that can be used to predict complex non-deterministic machine behavior (similar to how we already try to predict non-deterministic behavior, or not) in complex systems.
As a corollary, I would argue that machines have already modified humans. They affect the way we think and reason. They are an integral part of our life and the machine abstraction can be projected onto that which it does not describe completely (in which, the technology that exists now can not construct the technology we envision). We program machines, machines program us. This seems obvious.
The second AI created by humans is unlikely to have same goals or to be predictable by the first AI (or whatever evolves from it), thus I guess it's something that the first AI would want to prevent, either by eliminating humans or by fully controlling them (for their own good).
It's extremely important that the first AI we create is well thought out as we won't have the second chance to fix things up, once it's here it will have god-like powers over us.