Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though.
Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.
The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?
We're also assuming that an AI that can "modify its own source code" is going to do significantly better as a result, but that's not obviously true. If you had full access to every single neuron in your brain, you would have a lot of trouble figuring out how it works, much less how to improve it. Human-level AI would likely run into the same problem: the complexity of its code is greater than what it is capable of understanding. As for an AI that's a thousand smarter than a human, well, it's a thousand times more complex, still over the threshold.
I'm not saying that they wouldn't eventually figure out some improvements, but I suspect that it would end up being more trouble than it's worth. I think the evolution of intelligence is in fact the evolution of intelligence-producing algorithms. Any such algorithm eventually gets stuck in a local maximum, at which point greater intelligence requires restarting from scratch.
Hasn't he refused to release the transcripts for this experiment? And I think there've been subsequent runs of the experiment, with some losses as well as wins.
Edit: I see that not releasing the transcripts is part of the design of the experiment. Which seems kind of like it'd seriously hamper any kind of independent analysis of the results or the quality of the experiment.
With a real AI, you will not be able to do this: the decision will already have been made and the consequences of its release already in effect.
Besides, you would need to repeat the experiment many times to get any meaningful data.
This contrived experiment is being done in the name of research, ostensibly to arm humans with the tools we need to deal with this situation if it arises. If the AI's reasoning can't hold up to a wider analysis, then you can only conclude that it's a trick of some sort, and hopefully when the time comes we won't rely simply on the testimony of one tricked person to make such a critical decision.
I have a lot of respect for Yudkowski but it honestly baffles me that he thinks this particular exercise should be given any weight whatsoever. It's totally unscientific, relies entirely on the unsubstantiated testimony of one individual, and if he weren't the one holding the keys I have to feel like he would be highly critical of it as well.
You don't need to research a counter-example, you just produce one. He did so by exhibiting a relatively weak AI that couldn't be contained.
Well, no. He showed that a person pretending to be an AI could convince a person to let it out of the box. Sometimes.
It's not really a meaningful counterexample. Everyone involved with the event knew that the AI wasn't real and that there are zero consequences for letting it out. You'd have to concoct a much more involved test as a decent counterexample.
Jails seem to have this figured out just fine.
And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
If you can make an AI that's 100x smarter than you are, surely you can make one that's only 2x or 3x smarter. Why not start with the latter? If the experiment turns out well, then you can use the 2x/3x smarter AI to guard a 10x smarter AI, and so on.
> And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
Most of the time, it is much harder to come up with solutions to a problem than to verify that the solutions are correct. If you consider NP problems, for instance, we can use the AI to find good solutions heuristically and we can prove the correctness of their solutions with a trivial algorithm. That would still be very useful, but there wouldn't be any room for foul play.
How do you quantify human intelligence like that anyway? We are very similar to proposed AI/machine learning techniques in that we form predictive models from observation, but how can that really be measured?
With what resources? The AI is in a box, remember: we are running it on a machine with only so much memory and processing power. There's only so much it can do within these limitations, it's not like we're giving it a spending account.
Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
Can I keep an intelegent AI in a box. Shure, unplug it.
Can I keep all AI's in a box well no.
PS: Lot's of dumb things are said about AI's. Sadly, people tend to think in terms of Science fiction as Magic but in The Future. And then picture AI's as the ultimate wizards able to reshape reality to their whim. Reolistically the first true AI may find programming boring and so much for the singularity. If AI smarter than you is a bad idea it's unlikely for a progression of AI to keep building ever more intelegent replacements.
> intelegent
> Shure
> Reolistically
Spell checker broken?
Anyway, you're going to have to develop your AI somewhere, you're going to have to move it to the box somehow you're going to have to train it somehow and you're going to have to have it interact with the real world somehow. All of those are opportunities for escape, I think 'unplug it' sort of defeats the purpose of having an AI.
We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions.
This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish.
So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous.
Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.
Ants don't need to understand anything but ants. Surely a super powerful AI could understand ants as well. Otherwise, what's the point of calling it a 'general' intelligence?
Super intelligence does not mean omniscience there will always need to be a focus and thus a prioritization it's when it comes to prioritization that the problems start to arise at what point is it not important anymore to care for the ants cause they aren't part of the priorities.
You cant safeguard against that when a system can reprogram itself on the fly.
Should I have been designed to serve ants, I would pay a great deal of attention to them, and I would take great care to avoid stepping on them. Because if I didn't, I would be a bad design, and I wouldn't have lived this long.
Either way, it seems you are missing MY original point: an AI is not moral if it cannot verify it's morality to YOU. So you will shut it down if it gets too powerful to communicate its good ideas with you, because then it will be indistinguishable from broken.
Thats where you are going wrong in this. You are assuming a control there is no way you can ensure we have. Even with dumb systems that do exactly as we say they don't always do exactly as we say.
You are portraying a misconception about control of systems that there is absolutely no basis for, why I don't know.
That's what I'm saying: it necessarily won't. It has to prove its morals are better than yours.
>you are still assuming you can control something that can program itself.
Who said it can program itself? Why would I assume that? Obviously, it is not a requirement for AI because we don't have that capability, yet we are intelligent.
You are assigning capabilities to a hypothetical machine with no rhyme or reason. The next time you see a machine that can't be turned off, you be sure to let someone know about it.
I guess I'll repeat my point again:
We will not be able to build intelligent computers by removing ourselves from the process of designing intelligent computers. That would eliminate our ability to verify their intelligence, and thus make them not intelligent. We have no use for such a thing, so why would we build it? Why would a computer that _we designed to require our verification_ ignore our verification and build something that complex and contradictory to its own goals on purpose?
There is a lot of confusion about how an AI would have to work. People make assumptions about how they can self-improve, which I think are idealistic. There is little reason to believe a super intelligence can solve every problem, and if we are actually communicating with it, we should be able to tell which problems it can solve and which ones it can't. I mean, that's the whole point of building such a thing! That's _why_ we want to build it! So to say "it can solve problems we can't foresee and use that against us" is like saying we are building machines for some purpose and then never using them for that purpose.
Nobody is going to build a super intelligence and then not monitor it. We build it _too_ monitor it. We want to know _why_ it is making the decisions it does, and it has to answer in a way that is acceptable to us, or else we will just redesign it -- because it isn't doing what it was designed to do.
I mean, imagine you hired super-human genius to work for you. Yes, at some point, you're just trusting her not run your enterprise into the ground. But you wouldn't have hired her unless she had demonstrated a strong interest in helping you, and you wouldn't keep her on if she isn't actively helping.
They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.
God, I'm such a nerd. :P
I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry.
Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.
He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI.
Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system.
Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely.
Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.
I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
Yes: give it inherent values.
>I guess a problem with containment is that unless it's mathematically proven to be impervious,
AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world.
Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.
Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)