> Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world
Rest easy folks. This is purely theoretical.
Rest easy folks. This is purely theoretical.
AI Safety guys really like using the physically impossible to advance their arguments. It's bizarre! Pure fantasy isn't something worth reacting to.
The AI Singularity is like the Grey Goo scenario. An oversimplified model projected out so far in the future that its flaws become apparent.
I don’t need that version of the singularity to be worried.
I think in terms of “the event horizon” rather than “the singularity”: all new tech changes the world, when the rate of change exceeds our capacity to keep up with the consequences, stuff will go wrong on a large scale for lots of people.
As for grey goo? Self replicating nanomachines is just biology. It gets everywhere, and even single-celled forms can kill you by eating your flesh or suborning your cells, but it’s mostly no big deal because you evolved to deal with that threat.
However, thermodynamics says it becomes increasingly more difficult to distribute power (food) to a growing population and it hits a natural limit to growth, just like bacteria.
The visible universe is far too small to store that data. Like many exponentials too small. You can't even enumerate all the programs that work on a handful of inputs without running into the problem that the universe just isn't big enough for that.