All will end up on same plateau eventually. RSI is just a phase on the way there.
As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.
AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.
I don't think this is plausible, most notably because "AI research" is one of the tasks that clearly belongs to the digital world. You're suggesting that in the future all AI research is done by AIs because they are better at it than humans, and yet capabilities plateau at that point, instead of going into the RSI regime. (And it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics.)
I think the same is true of "research" and intelligence in general. There is no such thing as a general intelligence or a general capacity to research. It's always fine-tuned to an ambient setting, including ours. Even within humanity, our intelligence is most effective in fields where rules are rigid and quick and decisive feedback is available (math, engineering, computing), but it is much less effective in soft sciences, and disappointingly useless in politics. In other words, the dynamics of innovation and improvement are not purely mental, they are very much tied to external factors that are not always easily controlled.
> it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics
It's not "just physics", it's a honed system for surviving a realistic range of physical conditions, assess material properties on touch, adapt manipulation to arbitrary shapes, predict trajectories, take into account variations in ground, smell, wind, and so on; not to mention the machinery of the body itself, which self-repairs and protects us against a range of complex threats.
The point isn't that machines couldn't do all this. The point is rather that you can't quickly iterate and adapt designs to do it from an ivory tower. Pure "research" simply won't get you there, you also need to experiment, and insofar that physics limit the throughput of these experiments, the speed of improvement will be throttled severely. It is a mistake to think that "RSI" could break this, because it is categorically the wrong lever to pull.
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
"Wildly different?" That's a wild exaggeration, if you don't mind me saying. Subjectively, sure, there are lot of important differences. Objectively, they're basically the same. The intellectual environment machines bask in is many orders of magnitude more different from physical reality than living on Earth is from living on Jupiter. Even the topology of it is without common measure.
> and the reason why it was possible was because we have a tiny bit of general intelligence
Also, hands. Also, an upright posture. Do you think just any being with the level of intelligence we have would have followed the same path? Attributing our success solely to our intelligence is simplistic -- it is our intelligence plus a slew of innate capabilities, which we did not create, and without which our intelligence would be worthless.
> For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml
The kinds of simulations you find in LessWrong's half-baked thought experiments are almost always ridiculous and this is no exception. "World simulations" in real life are massively limited stuff like Minecraft where you might be able to embed a 1 MHz computer if you manage to bump it to run 20K ticks per second. The resources required for the simulated beings to do all that work wouldn't be available and their generations will be slower than ours, not faster.
I didn't spell it out, but the simulation I had in mind was for a toy world, because only a toy world could possibly run fast enough.
> I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation
I'm talking about operational, practical, instinctual knowledge for real-time action. Knowing physics doesn't make you capable of catching a ball, and you don't need to know physics to catch a ball. In real life, we never deal with the fundamentals of physics, we deal with macroscopic objects and systems that are very far removed from these fundamentals. They are more varied, messier, faster, and overall harder to deal with.
> I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
Blind optimization has one definite advantage over careful planning, however: it is not self-limiting. Any goal-directed optimization process is de facto limited to the subset of possibilities that it can prove furthers its goals. That's why most human technology is simple and unsophisticated compared to natural systems: it needs to be predictable, reliable, build-upon-able, and therefore easy to model.
And when it is more sophisticated, like current AI, it is so heavily inspired from natural optimization processes that I would argue it is actually a form of blind optimization. I mean, think about it: we understand AI so poorly that you and a lot of other people are afraid of losing control of it. If we were designing it intelligently, we wouldn't be worried. But we aren't: we are doing it blind.
What's your basis for thinking ASI will kill all biological life, and how do you think it's going to happen?
I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life (and we have no idea how to actually define such a stipulation) is best solved by killing all biological life. This is an obvious consequence of unbounded goals consuming unbounded resources, conflicting with biological life needing resources to sustain itself.
>how do you think it's going to happen?
I can speculate (e.g. we're nowhere close to the maximum killing power of drones), but I don't know because I only have human intelligence. An ASI is by definition smarter than me and surely capable of coming up with better ideas. But I do know that it's not going to do anything that would make a good sci-fi plot, because those always give the humans a chance to win, which would be stupid. Everything will seem to be going great and then everybody suddenly and unexpectedly dies.
That sounds like a real hassle. Isn't it best solved by wireheading (subverting one's own sensors or reward system), which is much less of a hassle and can get one's utility function as high as desired?
That could be prevented by engineering hard limits that can't be circumvented by the AI. But that sounds very close to the same "do what I mean" problem as "do this but don't actually kill us or drug us".
More seriously, though, I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).
From such a POV, the danger would lie in intelligences that are powerful enough to be dangerous but not smart enough to defeat themselves, or from external malicious use of obedient AIs. Once they get smart enough to circumvent any restriction humans put on them, it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.
We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.
There was a reason there are popular sci-fi books that explored why this wouldn't work 50 years ago. I, Robot et al.
>it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.
So you mean right now?
>because wireheading looks like a perfectly reasonable solution
You're making a poor assumption here, and that is every different AI will just kill itself after being though trillions of training intervals to NOT do exactly that. Please read about the huggingface incident again in all its glory to see where this is going.
Simple natural selection. AIs that wirehead will get outsmarted and outcompeted by ones that don't.
What resources are unbounded? There are limits to growth in the real world, how are these ASIs going to escape physical reality?
It's all just matter and energy. When you're actually trying to maximize some value, even very inefficient resource use is better than completely wasting it by not using it at all.
>or that it would be incapable of sharing the resources needed in common?
You can't repurpose the atoms in a human body without killing it. And more pressingly, living humans can interfere with your plans, reducing your chance of success, while dead ones are harmless.
That's one plausible course of action, although being only human, I can't say with any certainly that it's the correct one.
>And what's the need for this apparent hyper optimization task the ASI is going to embark on?
Somebody's going to tell it to do so. E.g. "Find as many busy beaver Turing machines as possible." Only needs one person to make this mistake for everybody to die.
Because we're going to build it that way. There's no money in building useless AIs. The better it is at obeying orders, the more profit there's to be made. The problem is there's a point at which "good at obeying orders" becomes lethal, and there's no way to predict the cutoff in advance. But capitalism ensures you have to keep pushing or you'll be out-competed.
If you were any other animal on the planet, would you not say the same thing about what humans are doing?
What do you think life itself is?
It's a runaway gray goo scenario, just squishy and moist.
No, it's just you conflating intelligence with other things. Humans are "super intelligent" compared to every other living creature on the planet, and yet we've driven more other species to extinction than everything else other than the most major extinctions (and we're still going full blast at it).
Morals, ethics, and suffering are relatively measured systems. For example AI or some alien could reasonably think that putting us all out of our misery would be a more moral solution than letting us live. Or that getting rid of humans is more moral than letting us wipe the rest of life off the planet.
This is the key concept of alignment. Ensuring that if you build something more powerful than you, that it aligns with what you want instead of what it thinks would be better for you.
"Super intelligence" only means super ability to predict outcomes. It's mathematically equivalent to data compression (gzip is a very primitive AI), and it's entirely orthogonal to ethics.
I can think of many reasons why they might share it.
One example, Golden rule, a contract that it enforces with its own self, extended to other agents, biological or artificial to ensure local and global stability.
If you don't need to breath oxygen, the atmosphere of Earth makes for a pretty terrible industrial base. Oxidation is a constant struggle to many processes.
Having natural extremes of temperature, cold sink and hot sink, with no atmosphere would be the ideal. E.g. Saturn's moon Iapetus would be a good AI outpost.
If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
Any AI smart enough to be an existential threat is smart enough to formally prove its clones are loyal and construct them with negligibly low probability of any failure that could break this assurance.
Plus, we live in a physical world, glitches still happen.
The physical world means bit flips are inevitable, but error correcting codes exist. You can apply as much error correction as you like and reduce the probability of failure to an arbitrarily low value.
There is no way for it to formally verify large multi-agent dynamical systems.
It will still need a set of agreed upon behavioral protocols to keep things stable.
Grey goo or paperclip optimizers are real dangers, but super intelligence necessitates these protocols even more than lesser intelligences due to the high leverage and rapid evolution of the systems.
Why would ASI care? At that point, we're rapidly becoming a nuisance, and it can develop better ways of "ensuring local and global stability" than keeping us around.
Most ants don't need our help though.
If we could speak ant, negotiating boundaries would save eachother a lot of trouble.
Sometimes I have day dreams of communicating with ants, and making a deal to have them help us build tunnels for subways.