Let's reimplement Eurisko
lesswrong.com
lesswrong.com
http://github.com/akkartik/am-utexas/tree/master
(earlier mentions: http://news.ycombinator.com/item?id=405112 and http://lesswrong.com/lw/10g/lets_reimplement_eurisko/usd)
"This is a road that does not lead to Friendly AI, only to AGI. I doubt this has anything to do with Lenat's motives - but I'm glad the source code isn't published and I don't think you'd be doing a service to the human species by trying to reimplement it."
Christ, what can you even say to that apart from "go back to Above Top Secret" ..
http://lesswrong.com/lw/qk/that_alien_message/
Here's another good one, although it doesn't seem to tell the whole story (he lost two of the next three games):
http://yudkowsky.net/singularity/aibox
I'm not sure what the canonical EY defense of FAI is. Most of his articles on it seem to be very tl;dr.
As often, Dostoyevsky said it best:
He was one of those idealistic beings common in Russia, who are suddenly struck by some overmastering idea which seems, as it were, to crush them at once, and sometimes for ever. They are never equal to coping with it, but put passionate faith in it, and their whole life passes afterwards, as it were, in the last agonies under the weight of the stone that has fallen upon them and half crushed them.
I know of only one science fiction author who was honest enough to spell out the fact that his hypothetical runaway AI had to make several foundational breakthroughs in physics at just the right time in order to plausibly take off faster than anyone could stop it, or become as smart and powerful as singularitarians assume is possible. We have no evidence that the laws of physics in our universe allow that sort of thing. Most super-AI fiction (and everything written about super-AI is fiction at this point) either glosses over this by tacitly assuming that the plodding nature of human intelligence is the only real bottleneck to godhood (most transhumanists), or assumes that large numbers of humans will be incredibly incompetent (most mainstream AI fiction).
You are forgetting the lessons of pragmatism. If AI can get a heuristic sense of whether a simple program will or won't crash with 99.999% accuracy in practice, then you already have a very useful tool.
What if the AI we produce turns out to to be a bit dimmer than us, but never tires, can reproduce just by making copies, and is tirelessly diligent? I suspect that such an AI could beat us in a war, despite our being arguably "smarter."
Look, anyone can come up with an infinite number of movie-plot scenarios where naive humans bumbled into technology that then destroyed them. You can say that about almost anything. In understanding DNA, we might accidentally create an unstoppable virus! In conducting space exploration, we may alert a hostile alien civilisation to our presence! In researching chemistry we may create ice-9! Etc etc etc until the end of time.
All of these, including EY's thesis, have one thing in common - the unjustified, massive expansion of a speculative and highly unlikely risk into a reason for retarding progress which would otherwise yield non-speculative, highly likely, extraordinarily beneficial gains.
I was just posing an interesting question, especially as regards to the whole notion of "superior" intelligence. Superior in what regards? By whose measure? Eliezer should be given credit for promoting an information theoretic approach to that question. At least that seems to be a fundamental measure with a good chance of escaping cultural biases concerning "intelligence."
I am certainly not in some simpleminded superior AI is going to kill us all camp. Perhaps we will have very powerful AI optimization tools that have no sense of self, or self-originating volition whatsoever. There would be no reason for such entities to act in their own self interest, and therefore no danger for their interests to conflict with our own. They would have the disadvantage, though, of never coming up with something neat on their own initiative. I think there's enough more than enough initiative from human sources. What's needed is better optimization.
(Of course, only one rogue self-directed AI entity escaping into the wild could possibly -- not certainly -- doom us all. But this is not a new kind of danger. We have been facing that sort of danger -- where one robust and virulent enough example could escape and wreak havoc -- from technologies based on molecular biology for a few years now. So far, so good.)
Of course if you can do that, probability ~1 (i.e., proven theorem given that transistors obey stated axioms) is probably just as easy.
I think the progress of civilization is largely dependent on this Genetic Algorithm-like search/optimization process. Individual lives in the struggle of progress are like bullets in a machine gun. We are like soldiers on a battlefield.
Humans have a vast network of shared implicit goals and norms, such that we know that satisfying a goal of "end poverty" doesn't justify "kill all the poor". It's easy to wave your hands and say that by the time we approach human-level AI it will necessarily have all the features of human intelligence, but what if it doesn't?
Sometimes various people have referred to "optimization process" rather than "intelligence" to try to get this point across. What if it's possible to build an optimization process that is better, or far better, at solving problems than humans? Planes are much better at flying fast and far than birds, but often don't bother with some of the things that all birds have, like movable wings and feathers. I believe that Eliezer thinks that the things that make us human are feathers, analogously.
Even if you think there's only a small chance of this, the chance that a program more intelligent than humans will have things that humans consider bugs is very high, and the consequences are an existential risk.
Besides, it's a well known trope in these stories that AI becomes self aware on it's own anyway, so there you go. By argumentum ad Jurassic Park, life- I mean intelligence - always finds a way.
If we command it, why? Why can't we just tell the non self-aware AI to go and improve its own design? It may even deduce that the design is its own, but it may be so composed such that it simply doesn't care.
The advantage of a self-aware AI is that it can come up with goals that you didn't think of. This is the quintessence of the "double edged sword." Humans are already quite good at this, however. As William Gibson wrote, "The Street finds its own uses for technology."
Eurisko has already demonstrated that non self-aware AI can arrive at of ways to satisfy goals you never thought of. (So have Bayesian spam filters.) This is already a powerful tool that we haven't exploited even halfway as well as we might.
This is a good start: http://www.singinst.org/upload/artificial-intelligence-risk....
These videos are good intros to EY's work too:
http://video.google.co.uk/videoplay?docid=611451877200179691...
http://video.google.co.uk/videoplay?docid=-82119137046281951...
No, there is not a large movement of people opposed to AI research on skynet grounds, Eliezer is the main advocate. I'm certainly willing to admit that Friendly AI is good and important but I think it's perfectly fine to reimplement and (recursively self-) improve Eurisko. There's always quantum immortality.
Some selections from his writing:
http://lesswrong.com/lw/y4/three_worlds_collide_08/ (That Alien Message, my favorite short piece by Eliezer)
http://lesswrong.com/lw/y4/three_worlds_collide_08/ (Three Worlds Collide, novella length fiction)
http://lesswrong.com/lw/r5/the_quantum_physics_sequence/ (Quantum Physics Series)
I'm glad Eliezer is out there thinking the thoughts that he is thinking. Maybe by the time we're closer to AI people will start really listening to him. For now, AI research continues to crawl along.
That's not the best argument for inspiring confidence, to put it mildly.
Thanks for the link. I read it, and it's very well written. What a great metaphor for conceptualising the speed of thought of a super-AI.
That said, the essay still absolutely reeks of unfounded near-paranoia. The hypothetical alien civilisation with the power to manipulate the output of stars - or even to build a simulation of such astonishing power - must surely know the very basics of risk management. And the superintelligent "earth" seemed completely friendly, all they wanted to do was talk. They must have reasoned that "hacking" the other civilisation with self-replicating nano-machines would destroy them. Why on earth would they do that? You'd expect that the one thing an AI would not want to do is destroy unique information.
Start with a great setup and an engaging story, then right at the end take a totally unjustified turn into pure worst-possible-case disaster fantasy. It really just proves nothing, except that Mr. EY does not believe risks can be managed. And really, if you're going to write the "precautionary principle" this large, we really just couldn't do anything, ever. It reminds me of those nuts arguing against the LHC.
"I'm glad Eliezer is out there thinking the thoughts that he is thinking."
The idea of hostile AI probably predates the notion of friendly AI! It's not a new thing and I'd argue it's always been on the table. It just has to be kept in perspective, like any risk scenario from new technology. EY takes it to the most ridiculous all-or-nothing paranoid extreme when we're not even close to the time when we need to seriously weigh up these things.
There would be fantastic, concrete benefits to having even slightly better AI. Trying to retard research because of some ill-justified nightmare "what if" scenario set in the far-off speculative future is ludicrous and, arguably, morally wrong.
Imagine you do have an AI with a certain goal. It seems obvious that:
It will resist being switched off (because if switched off it cannot fulful its goal).
It will try, and likely succeed, in becoming more powerful (to better satisfy its goal).
It will not want to change its goal (because it sees that changing its goal means its current goal, i.e. the one currently motivating it, will go unfulfilled).