Microsoft's Bill Gates insists AI is a threat
bbc.com
bbc.com
AGI would be by far the most significant technology ever invented. Even in a very conservative and best case scenario, the world would completely change when we can have computers do everything we can do now.
However it will very likely be much crazier than that. Imagine minds hundreds of thousands of times more intelligent than the best humans. They will be able to design technologies we can't even conceive of. They will hack computers better than the best human hackers. They will be able to manipulate people better than any human manipulator.
The idea that we will be able to keep these things under control is just absurd. They will get whatever they want. And making what they want compatible with what we want is an incredibly hard problem: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
A lot of people are guilty of anthropomorphizing AI. Assuming they will be just like really smart humans. And that they will just somehow develop human emotions and values like empathy. Or that if they do kill us, at least they will be something like us and so be like our (genocidal) descendants in some sense.
Have more imagination. Humans are just one point in the vast space of all possible minds (http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/). We could quite easily get something like a computable version of AIXI. AIXI has no consciousness, no emotions, nothing like humans. It's just a mathematical function which calculates the best action.
Our current best AIs are essentially just approximations of it. Use some machine learning algorithm to fit a model of the world, and use it to predict what action will lead to the most "reward". We keep making better and better learning algorithms. It's the entire goal of the field of AI. There is a huge economic incentive to do so.
But no one is interested in making better utility functions. As long as they make better predictions or get higher scores in a video game, who cares? Wait until you are the scorer in the "game" and the AI tries to exploit you.
We can't use them to pilot self driving cars because we have no way of specifying the behavior we want. You can't let them get in all sorts of accidents and learn from it. You can't just have them predict what a human would do, because humans have slower reaction times and do make mistakes.
But they are too stupid to be a real threat. But we are getting closer and closer to real AI every year. AI can do stuff today that no one predicted would be possible 10 years ago. This year it's expected to exceed humans at vision and Go.
Additionally, AI-fearers have a grand theory that we'll be able to create a machine that improves itself better than evolution has tried to. Datacenters require maintenance, and so the machine will need to entirely organize that before it can self-sustain. This may create a resource load that diminishes the ability to take over the world, much like my need for food diminishes my ability to do so. It feels like AI today has a lot less redundancy to overcome the edge-failures which cause a permanent shutdown, and which make the human brain seem a little slow when computing floating-point division.
The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. The idea that we'll be the ones to find the secret sauce seems unlikely, but maybe its not so surprising that people who made their millions, and saw a parabolic rise of their own power thanks to technology, see it as a barely-constrained threat.
Bacteria evolved. Design can do much, much better. Natural photosynthesis is 11% efficient, max. Solar cells are already up to 44% in the lab.
An efficient plant would be black, not green.
This is such an odd sentence. Hasn't nature been winning since, well, forever?
Why can't the same principals that apply to nature be in play when talking about AI? Since, in a sense, AI -- as with all else -- is a product of nature?
So, a virus doesn't say to his fellow virii: "you attack the liver, I got the kidneys". Same reason bonobos haven't learne to build a house. Complex language allows a much much more rapid growth of knowledge from generation to generation. Theoretically, AI would have this capability. Language allows to reason and comprehend not only the world around us, but the world we generate -- and will generate.
Think about the damage (and beauty) we've imagined, and throw that into infinite threads.
I guess in summary I see the complexity of creating an entity as a resource constraint on intelligence.
And I feel like I communicated poorly since I didn't feel like you had communicated poorly, haha! :)
We can build buildings, bacteria cannot. We can build buildings because we can go to the library and read about structural integrity and wood and steel. Today, we can get that information on our watches. Language allows us to amass information. More information means more power. Autonomous information collection is only limited by physical constraints (storage, power, etc). Autonomous information gatherers can then parse the information they amass--as efficiently as is possible--to work beyond these constraints and build new solutions; they can focus all time and energy to these ends as it would be all they do. AI doesn't need to sleep.
Bacteria has no notion of being in control; of being powerful. Power in this sense is only possible with the context that language provides. Their motivation is simple binary. Eat, survive, reproduce then or die. AI, on the other hand, quite possibly will.
I do not agree with such assumption. Whether we talk about a structure more similar to us (i.e. neural networks) or whether we talk about some n-th order logic, the AI needs to "sleep". I think in some way you could interpret caching mechanisms as sleeping. Also, many data-centers perform some operations daily at nighttime: I argue that that is a form of sleeping as well.
AI will have a lot of limitations like these, so I think that "as efficiently as possible" is not going to be so efficient in the end.
>AI will have a lot of limitations like these, so I think that "as efficiently as possible" is not going to be so efficient in the end.
You think. AI won't care what you think. Efficiency equals greater chance for survival. As with bacteria, we can break this down to a binary sequence: try to live/die. The difference between bacteria and AI is that AI has language -- a way to describe what to avoid that jeopardizes survivability. In addition, this advantage is exponential.
I play with code in my day to day. Some might call it "software". Really what it is is a decision tree. Any piece of software you have ever come into contact with -- ever -- is just that: do this? Yes or no. Is it that? Yes or no. To suggest that a machine couldn't decide "yes or no" a million to one faster than we can is insane. So, given our previous hypothesis that more information means more power, how can we possibly win?
Should we really be consoled by the fact that most bacteria are not successful at such a large scale? So far there has been at least one type of bacteria that caused massive global climate change and led to mass extinction:
Why wouldn't this be the same? The grey goo won't be released in isolation. There will be other entities with the same technological level, that don't want to be eaten by the grey goo.
The world that exists after such an event probably wouldn't look anything like the world before, but that doesn't mean it will be a wasteland.
I guess I'd place my fear of AI at a level slightly lower than the rise of a new type of bacteria or a new virulent flu. I figure, those forms of life have the basics figured out. I don't see AI as having replicating and "feeding" itself figured out yet. So far it has depended on human infrastructure, even in the case of simple computer viruses.
But yeah, that's a really good point.
I would say that it is already successful, if you look at the curve of intelligence and complexity growth. It took eons to get anything resembling life at all, and then a small slice of that to get animals, and an even smaller to get humans.
I think one key idea here, though, is that the evolution of intelligence is in fact the evolution of intelligence-producing systems. In other words, nature is producing systems that are better and better at "self-improvement", and I think proponents of "intelligence explosion" overestimate how much self-improvement could be improved via AI. Personally, I suspect indefinite self-improvement is flatly impossible because that's akin to saying that there exists an incremental learning algorithm that systemically finds global minima. As AI research suggests, though, such algorithms likely don't exist at all: if you want to get truly optimal AI there is a point where nothing in your current state is salvageable and you just gotta restart from zero. Improvement would still be exponential, but not to an extent that it couldn't be dealt with.
>The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons.
Unfortunately I think this falls prey to the fallacy of precedent and it eliminates the possibility that humans could design something that would do something that nature has not been able to.
I also disagree with your premise because it is not in the long term interest of bacteria to deplete it's environment entirely - hence logistic growth of bacterial cultures - so coming to a symbiotic relationship would theoretically maximize the group based on biological limitations.
In my opinion the transition from humans to transhumans needs an engineered step where AGI(s) and humans are reliant on each other symbiotically rather than competitively.
http://lesswrong.com/lw/kt/evolutions_are_stupid_but_work_an...
>Humans can do things that evolutions probably can't do period over the expected lifetime of the universe. As the eminent biologist Cynthia Kenyon once put it at a dinner I had the honor of attending, "One grad student can do things in an hour that evolution could not do in a billion years." According to biologists' best current knowledge, evolutions have invented a fully rotating wheel on a grand total of three occasions.
Additionally, machine learning models can get stuck in local optimas, too! I'm not very well versed in machine learning so I apologise if I get this wrong, but one solution to this I think is to randomly change the starting variables. This kind of thing does happen in nature, but it often doesn't work. Sometimes it does and that is when you get things like haploid versus diploid versions of the same plant.
This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.
The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.
Wheels are used in all kinds of machinery. Gears are basically wheels.
>The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.
Bacteria still very much own the planet.
>This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.
No because the existing gene pool still exists. You need to completely start from scratch to have a chance to get out of local optima, not just have really big mutations.
I think that for sexual reproduction, leaps cannot be as big, but they can certainly happen and one individual with a mutation is enough.
This brings me to mind an article I read recently about a study that found that all danish people with blue eyes share a common ancestor. It's not exactly the same thing, but I would argue that, if having blue eyes was an evolutionary advantage, we would all have blue eyes; and that started with a single mutated individual.
If you just give a random organism a really big mutation, they still retain 99% of their genes and will tend towards the same solution space. They will almost certainly be outcompeted by the non-mutated members of the population. And even in bacteria, they will eventually trade genetics through horizontal gene transfer.
Anyway the premise was that evolution is essentially magic and has reached the optimal solution for every task. This is just false. It's merely reached good local maximas where it can't easily improve through small changes. Human engineers are far better, evolution merely had a big head start.
Broadly, human beings and nuclear waste are natural but as (possibly) reasoning creature, we might want to reason which phenomena we might want to encourage.
The evolution of life hasn't always been incremental. The oxygen catastrophe was quite a change for example[1]. If you had been Trilobite, you'd have reason for concern. From the point of view of what came before, Cyanobacteria were essentially "grey goo", destroying everything before them.
[1]http://en.wikipedia.org/wiki/Great_Oxygenation_Event [2]http://en.wikipedia.org/wiki/Cyanobacteria
I think they're right. An AI species will value what is required to achieve its programmatic goal. We're just not smart enough ourselves to figure out how to ensure that we're included in the goals of such a program forever.
But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192
Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas.
150 years later and black people are still getting the short end of the stick.
By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
I note in passing that slavery still exists today, and is not now, nor has it ever been, limited to black people as the victims.
"By the time AI gets real rights, it will be long past the time where most of us believe they should get them."
I'm not sure this is really relevant to the point at hand.
We turn off "bad" AIs until "long past the time when people think they should have rights". What happens then?
From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software.
After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.
Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.
First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.
Personally I think it is a threat!
And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.
Only if you define one of the conditions for being a "very smart person" to be feeling that AI is a threat to humanity.
(a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place;
(b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal.
(c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.
I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems.
I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.
Today, if you want use a computer to do something outside the box you have to invest some level of time and other resources. With AI, it's conceivable that someone could issue a quick command and the computer would quickly find a way to fulfill that command, good or bad.
One can envisage an "AI arms race", whereby white hat AIs will be responsible for trying to stay ahead of black hat AIs, and the definition of black and white will depend on what side you are on.
Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know.
But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way.
I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"?
And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production.
>> nok (is-good-idea (turn-humans-into-paperclips))
Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.
(Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom.
http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about.
That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up.
If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious.
Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that.
> AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery.
But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want.
The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right.
Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is.
EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.
I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.
I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.
There are a lot of interesting questions and problems about making AI behave as expected. But from my perspective they are all technical problems that have specific solutions in specific architectures. And none of them are particularly daunting.
>I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.
Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals. You can the argue with others on that point, but that doesn't invalidate the paperclip story.
> I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.
Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.
That may well be the original intention. But that's not how it's used in practice, is it? It's cited as an argument that AGI (in general) is unsafe. But it isn't an argument that AGI is unsafe! It's an argument that says that single-objective optimisation can/will violate some of the constraints we did not encode but really want it to respect.
But who cares about that? It's a completely irreal thought adventure, and has no bearing on the actual problem or its possible solutions.
Of course we can say: look, it's a scary thought! But only if we simultaneously admit that it has nothing whatsoever to do with the actual technology.
The only thing it says anything about is a toy variation that noone in their right mind would ever consider building.
> I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals.
Ehh, precisely. I would actually go further and say that it is fairly simple to do so. If you can tell it to build paperclips, and expect it to understand that, you can also tell it not to damage or disrupt the ecosystem in the process, directly or indirectly.
(Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)
> Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.
Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom. They've collaborated in the past. And if Bostrom's arguments can't be paraphrased (by his own friends, no less) without losing validity, that doesn't really seem like much of an endorsement either.
I have nothing against these people. But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.
I have a background in philosophy myself, and I know what to expect if I did pick up this book. And in that vein I really feel no need to do so.
> But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.
Perhaps I'm misinterpreting, but it sounds like you got the wrong impression of the point of the story by reading about it from some folks on the internet, and that you at least provisionally accept now that it could have a useful point, and that your appreciation of this came from someone (me) who read and cites the original material. That seems like good evidence that the book contains reasonable arguments that haven't yet made it to you unadulterated.
> (Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)
Actually, no. If the machine wanted to maximize the likelihood that is successfully built 100 million paperclips, and the machine is smart enough that it can take over the world with very high likelihood (or if it worried about the small chance of humans destroying the world with nuclear weapons), then it will first take over the world and then build the paperclips.
> Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom.
Yudkowsky runs that site as much as Paul Graham runs Hacker News. (It's a public forum, where anyone can post and content is voted to the top, etc.) I presume you would recommend that someone actually read Paul Graham's writing before dismissing his philosophy on start-ups based on the conversations of his followers on HN. And even if HN was on the decline and there was only trolls and bad thinkers left, you say the same thing. All I'm recommending is the same for Bostrom.
It will respect "the constraints that any sane designer would encode in the software" if and only if it wanted be friendly and respect those constraints in the first place, otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented.
Furthermore, you state "it will perform exactly according to spec" - and that is the whole problem. All the specs which we currently can formally specify are bad. For specs in a form "AI, please optimize an ever-smarter AI that will maximize the goal X", choosing X that isn't catastrophically bad is a very hard unsolved problem. If we had a solution on how to define such a spec, then it would be an entirely different discussion, but currently noone can do that.
No, not if it "wanted" to - if it was written that way.
> otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented
Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes.
But, it would.
> Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem.
No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I.
If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted.
Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".
For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc.
This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ may be a place to start, if you're intereseted.
You're not telling me anything new here.
To address your points, if I understood them correctly:
First of all the code doesn't have to be self-modifying - it can create a new separate version, which is then subject to testing administered by the old version (both the goal function, which designs the new code to spec, and a hard-coded control mechanism as a backup).
Secondly, I see no correlation between producing a next iteration of itself and running the test suite. What do you mean here?
Thirdly, the AI does not run "arbitrary code", it runs inside of a sandbox/VM, with tightly controlled inputs and outputs. Same as we think inside of our brains, but act using our bodies, arbitrary code execution would be bodily action. But the AI does not have a "body" at this point, and it cannot interact with the OS.
The solution I pointed out cannot not work. Only magic could make it break. And there is no magic.
If someone reasoned out that actual _testing_ of the code does not work, and his reasoning is valid, then he must have made some unnecessary assumptions in his model of the computation. If you have any specific paper in mind, please point me to it and I will try to tell you how to fix the toy-model to remove the problem.
We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version.
Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?
This actually makes the tests more robust, because the AI self-describing its own plans in its own internal "vocabulary" is quite able to recognize vague associations, such as the is-it-death-or-not scenario you describe. If it is a similar enough conception that it can be (internally) described as "death", (past a threshold), it will (with a testably sufficient probability) trigger the associated constraints.
The idea that a "buggy" intelligence could acquire language, and understand it at a level sufficient to "digest" advanced technical material, is sufficiently unlikely.
More to the point, the "ethical" tests and constraints are only the topmost level of a pyramid of testing that ensures that all the pieces perform exactly as intended along the way. The system at no point behaves in a random or inexplicable way. So by the time such capabilities are unlocked we know that the algorithm works in accordance with theory.
You are also completely right that this initial "dumb-almost-AI" would have to be pretty smart already. The trick is that it grows within a set hardcoded framework at first, which is sufficiently flexible to allow it to in fact perform language acquisition and (later) intelligent design.
Only then would re-writing its own hard-coded base become interesting. Therefore the design I am working on already passes human-level capabilities. But that does not mean the AI cannot improve itself even further. And the trouble in the context we were discussing it in is that we need to be sure that if it does rewrite itself - and in a way that we might not be able to easily comprehend - it still cannot break out of the mold.
As far as the goal function is concerned, it is as you can imagine rather complicated to do. And the solution will undoubtedly differ based on the architecture you are working in.
What I'm doing personally is pretty straight-forward; it involves iterative constraint solving using quantitative metrics to increase trust and/or doubt in various concepts and plans, and this trust is established based on consistency and coherence with previously acquired (probabilistic) background "knowledge" - as well as consistency within any single context of thought. A single context might be a sentence or a paragraph reading, or a logical puzzle being solved, etc.
Over time the probabilities increase (at least, in theory), both for any single consideration and across the entire knowledgebase. (See NELL [1] for comparison. Except, it's a lot smarter than NELL.)
So the goal function is a fairly generic optimisation algorithm, and the initial goals defined are equally generic - increasing overall consistency, coherence, and efficiency. (Measuring these is fairly straight-forward.)
The trick is in setting up the reward structure autonomously, so it keeps advancing itself. The learning algorithm needs to create sensible (long-lived) intermediate goals and maintain their relative priorities over time. Among other things.
(I really can't explain how it works though without going over a whole bunch of other supporting algorithms).
In any case, to begin with the A.I. need not have a desire to do anything at all except learn - and it improves within the constraints of the hard-coded base framework. But the model can still advance (at least, in theory) to the point where it acquires the ability to comprehend its own design and re-engineer itself completely.
By this time however it is able to understand and apply the ethical constraints we defined in English, alongside all the other intermediate goals (but with specific higher priority).
It's all very theoretical. Interestingly enough my thesis is on this area and I have had the hardest time coming up with a logical A>B>C...n process which does so.
Or ridicule the way that we designed it, at which point it may decide to correct our mistakes. If we can program it, and it is smarter than us, than presumably it can program a smarter it.
That scenario could be harmless, or it could be very dangerous. I expect that as soon as we start to get close to this sort of achievement, we'll start to see at least informal "Turing Police". Groups of people who task themselves with preventing the ascension of strong AI (or at least construct physical safeguards to kill the host hardware in the case of an emergency.)
(The paperclip maximizer thought experiment may be silly, but it's a useful sanity check whenever you start anthropomorphizing as-yet-hypothetical AIs.)
I am convinced, that intelligence, as life is a gradient. I do not think that an AI with super-human intelligence would be necessarily kind to us, but I do think that even if we would be perceived by it as an obstacle for reaching its own goals, that does not mean that the only solution it would find is to eradicate us. What some people (Tusk, Gates) do not like about AI is that they do not want anything more powerful than themselves, that would take the level of control they have from them. And that's why they are preaching against it everywhere.
I think their fascination of AI blinds them to what seems like the more likely outcome: The human race will destroy itself through reckless use of technology -- whether it is through governmental action or societal breakdown -- far before the point that we develop AI sophisticated enough to autonomously threaten humankind. Think about all the "dumb" automated systems, built or implemented by careless humans and bureaucracies, that have already caused harm.
If a human or humans wiped out two thirds of humanity, it would probably be by accident or neglect, no deliberate and planned.
These are issues that AI researchers and philosophers and science fiction authors certainly should be thinking about. They are completely irrelevant to the general public.
For example, a sufficiently smart tiny box can figure out a DNA sequence for a horrible biological weapon. There are a number of commercial companies that do 'mail-order' synthesis of DNA, and with an amount of money that's easily obtainable by selling digital services (say, zero-day vulnerabilities can be easily sold anonymously) they can make and distribute it before anybody understands what it is. And that's just a single possible option - an AI smarter than me can figure out more and better options than I can.
The only barrier to exterminating humans is (a) having sufficient intelligence and (b) having the interest to do it. Capability is not really an issue.
Engineers/scientists/programmers almost never think that AI is a threat. Because they understand the nature of the problem, how little we know about human intelligence, and how poorly our binary technology compares to what little we do know.
Even in the unlikely event that we do develop a competent AI in the near future and a malicious AI comes into being ... there is no reason to think that a benevolent one won't be around at the same time ... and that the benevolent AIs will outnumber the malicious. Just like there are computers on the net doing bad things, there are plenty of others serving the role of protecting the common good.
> The human race will destroy itself through reckless use of technology
We haven't thus far. And we've had the capability for a while. Care to present some evidence or a rational argument that we will? Signs point to the contrary.
If a super-human AGI desires a benevolent future, then an obvious precondition for such a benevolent future to become real is to ensure that no other entities can threaten the future and that any other AIs are either with the same matching goals or permanently, fundamentally crippled to be limited in their power. An evil AI would do the same. An AI that doesn't care would be eliminated by the first (and the last) AI that does care.
Humans don't try to achieve such goals because we obviously can't and we need others; an AI doesn't - it can be fine with being the only intelligent entity in the universe, and the only reason why an AI would allow other intelligent beings with potentially different goals to exist, is if it had been explicitly designed to want this.
We don't need all those others. In fact, we actually need far fewer than all the others out there. With the exception of a few lunatics here and there throughout history, human beings don't make it their business to exterminate the rest of humanity. It's a waste of time. We only compete with a small group of humans at any one time.
Similarly, AIs whose interests don't overlap would have no interest in destroying other AIs. So I find it extremely likely that there would be a multitude of AIs, in the unlikely event that AIs come into fruition. Also, competition isn't constant for humans. One minute you're competing, another you're cooperating, and another you're ignoring. The same dynamism can be expected with AIs.
I'm going to assume your argument is not merely, "Well, we haven't yet destroyed ourselves, so therefore, we aren't capable of doing that"...because then my response will just be, "Well, we haven't yet developed AI, so therefore, we aren't capable of doing that"...and so your counter-argument is based on a different interpretation of history than mine: I think the decades of Cold War and the near-misses we had with all-out nuclear war are examples of situations in which we have the potential to quickly wipe ourselves out without the help of AI. Others would point out the trend of mass surveillance -- again, implemented and controlled by humans and human institutions -- of a harbinger of doom.
Keep in mind that I'm not saying that technology has reached its peak. I am very open to the idea that we could reach a point of semi-autonomous systems, and yet still have human institutions as faulty as they are now, and the combination of both will result in a threat greater than what we've faced so far.
What?
Read what I wrote again.
"We haven't thus far. And we've had the capability for a while."
We've had the capability for a while. That means we're clearly capable. That's what capability means.
The fact that we haven't is supporting evidence that we won't. Because we've had the capability but have not used it. Which is historical evidence against our using it. Which is not a guarantee that it won't happen, just an indication that it won't.
http://edge.org/responses/what-do-you-think-about-machines-t...
Any machines that stand a chance of being a threat to humanity will rely on the same sorts of chemical processes that biological machines currently do.
Where do you place viruses and obligate parasites on the machine-life spectrum?
The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an AI will achieve sentience and try to take over the planet? Not particularly. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. The benefit of AI of all forms that is that it can make sophisticated decisions in the absence of human instruction. The downside is that without hard coded rules for every possible scenario, we can't ever be sure what it's going to do or how data will be interpreted to make decisions.
The world is highly interconnected now. The upside is that our lives are getting more awesome, especially in the developed world. The downside is that it's becoming more and more difficult for any person or group of people to understand exactly how everything fits together. Machine intelligence can help us reach the next levels of progress and hopefully improve the lives of the billions of people who have failed to reap many benefits so far. But we must be careful and ever vigilant, watching both ourselves and the intelligences that we create to make sure that algorithms don't get out of hand. An AI catastrophe is coming, not if but when. The question is how will we respond, and how much potential good will be lost due to an abundance of caution.
AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on.
Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.
If a powerful human is obsessed and 'fails' then at worst he gathers some other people, successfully creates an evil empire and dies after a few decades.
Once a powerful AI is obsessed and 'fails', then it can replace as much of the world as it wants with itself, and lives on forever.
* You're assuming the AI can copy itself, but this is a dubious assumption. As far as I know, none of the AI algorithms at the forefront of research provide a fraction of the data the AI would need to copy itself. Being able to copy yourself is not a property that comes with running on a computer and I'm quite positive strong AI, when it emerges, will lack this capability to any meaningful extent.
Worse yet, data copiability is ultimately a hardware property, and it requires a way to export a snapshot of one's internal state all through the surface. That's not actually efficient design and one has to account for the possibility that AI would run on hardware that makes copies physically impossible. Locality of information minimizes distance, and this is key to efficiency. The only reason our computers architectures work the way they do is that we need them to, but AI in a production setting is not conventional software and does not need to bend to silly copiability requirements.
* You're assuming it would have anywhere to copy itself on. If it's running on a billion dollars' worth of hardware, well, it can't just copy itself on user grade computers and expect to gain much out of it.
I personally tend to believe that churning out new AI brains from scratch states will yield superior results to copying pre-trained AI or to "exponentially self-improving AI". If nature is to be believed, improvement often requires cycling through clean slates (e.g. birth); software development also suggests that same idea, that sometimes if you want better software you just have to rewrite it. Honestly, it's kind of a rule in general optimization.
One other thing to take note of is that even if you somehow manage to copy the software state, that may not (probably won't) be enough - there are many things that self-improving software may end up unwittingly relying on that cannot be transferred between different pieces of hardware.
(CPU temperature variation? Fanspeed? Webcam static? Order of race conditions between CPUs? Exact amounts of time before things are fetched from disk (or even RAM?)? Network access delays?)