Can we hurry up and create strong AI already, before someone gets everyone killed?
EDIT: Somehow I didn't expect that offhand ha-ha-only-serious comment to kick off a massive discussion about the capabilities of AI. Fun, but not my intention.
North Korea not having a nuclear weapon is so much more important than nobody having one, that the two discussions have virtually nothing to do with each other.
It's not one is just a slightly greater degree than the other, they are so different they are not even in the same ballpark.
And what does having just one planet have at all to do with nuclear weapons? I'm not seeing the connection. All the nuclear bombs on earth would do nothing whatsoever to the planet, and barely anything to the inhabitants except for a minority of them.
And then the sudden jump to strong AI? Hu? That came out of nowhere.
You imply that strong AI has some sort of connection to "everyone killed", first of all, how exactly would everyone get killed anyway? Humans don't have the ability to do that, not today.
And strong AI would, if anything, make it easier to do that, not harder.
So your post really is quite perplexing if you actually read it.
Not to mention that nuclear winter is a very real (though unproven) concern.
No, that is simply not true. Not in the slightest.
Five countries detonated 2,000 nukes over 35 years. You could simultaneously set off every existing nuclear warhead inside of Wyoming and it would have a negligible health impact on most of the rest of humanity. You couldn't come even remotely close to killing everyone with nukes, unless the global arsenals were vastly expanded; the US alone has 27,000 cities. Russia's entire stockpile could kill at worst 1/3 of all Americans, assuming they all worked properly and hit their targets. The notion that nuclear war could end the human race, is entirely a myth (which is not to downplay the actual damage such would cause).
The desire has always been to make sure everyone is properly afraid of a nuclear war, because of the destruction it would indeed cause. It's desirable for it to be considered off the table entirely, a true last resort response. There is a large gulf between an outcome that destroys all of the relevant cities in the US and ending the human race. The 'mistake' is an intentional over-estimation of how much destruction the whole can cause (Why? To properly cause immense fear, so global nuclear war is thought of as a humanity ending event; and that's probably sane reasoning). Back in reality, only a few nations would be heavily destroyed, and the amount of radiation most of the rest of countries would absorb wouldn't matter in regards to survival.
Some will claim it would cause nuclear winter, destroy the global food and fresh water supply etc. It wouldn't come close to causing that. Nearly the entire planet would remain fundamentally just fine (the nukes would be focused on urban areas anyway). We know for a fact that large numbers of nuclear detonations (eg 178 in 1962) don't just magically spread globally and start killing everyone via destroying the food supply or other similar mass outcome events. Specific small areas of the planet would not be fine of course, they would be completely destroyed. An extremely small percentage of arable land could be destroyed, even if it was specifically targeted. Within less than a century you'd never know a global nuclear war occurred as far as obvious signs in nature are concerned, outside of the specific areas that were hit.
That aside, widespread radiation would ruin significantly larger areas (nuclear test sites intentionally avoid areas and conditions that would allow the spread of fallout, whereas bombardment would not); destruction of numerous functions of society and common services would kill many more.
Whether some subset of humanity would survive or not, this at the very least seems like a scenario well worth preventing.
Take out the big university cities, big finance cities, some semiconductor manufacturers and some ports and we back into victorian era.
Many of the folks thinking about and working on solutions to existential risk consider AI pretty much the universal solution (and, at the same time, consider unfriendly AI one of the notable existential threats).
What makes you think that would happen?
Be logically consistent please, either people will not set off nukes, and also program AI "correctly".
Or they will set off nukes and program AI "wrong".
In either case AI has nothing to do with safety from nukes.
Strong AI is not a God, it does not have the ability to do the things you are imagining. Among other problems, what makes you think those things are even possible in the first place? No matter how good the AI, it still has to follow the laws of physics.
I certainly hope the set of people with access to nuclear weapons would not be the same people building AI, or we'd indeed be quite screwed.
> In either case AI has nothing to do with safety from nukes.
> Strong AI is not a God, it does not have the ability to do the things you are imagining. Among other problems, what makes you think those things are even possible in the first place? No matter how good the AI, it still has to follow the laws of physics.
The laws of physics would govern the computational substrate on which everything else runs; what runs on that substrate can emulate anything it likes. The major assumption I'm making is that we can successfully build strong AI (AI capable of self-improvement past human), which seems plausible but not a given.
Well, in fact they are, (or would be) the exact same people. But despite that we won't be "screwed".
Or did you think that if strong AI showed up the Governments of the whole world would not rush to have their own?
> what runs on that substrate can emulate anything it likes.
At what speed though? It's not clear that an emulation in a computer would be anywhere near as fast as the actual thing (i.e. a human), so it might not actually be possible to upload people digitally. (If nothing else, the speed of light limits how fast you can run a CPU.) It's also not so obvious that it's even possible to copy a brain, never mind running one.
All your assumptions are predicated on the ability to emulate a human in software, if that's not possible then none of what you wrote about strong AI will happen, instead people will continue to be people and AI will just be another thing to ignore just like we ignore all the marvels we have now.
> The major assumption I'm making is that we can successfully build strong AI (AI capable of self-improvement past human), which seems plausible but not a given.
An even larger assumption is that it's even possible to do that. Never mind can we. Can an intelligence program something smarter than itself? I don't think so. I don't think the AI can, I don't think humans can.
So far the only known method of making a smarter version of something is by random permutation and picking the best one. It's not an accident that all AI used today is by that method. I'm not aware of any that are programmed from first principles.
Governments can't "rush" on the same scale an AI can, and a strong AI with an appropriate value function wouldn't permit the existence of any other strong AI that doesn't share its value function (since such an AI would fall somewhere between "counterproductive" and "potential existential threat"), nor would "of their own" have any meaning unless that value function was hopelessly broken.
There shouldn't be a "second" strong AI, only a first.
> It's not clear that an emulation in a computer would be anywhere near as fast as the actual thing
Humans operate relatively slowly on the scale of computers; we're just massively parallel and have a fundamentally different architecture. That's not at all insurmountable even for human-built technology, let alone technology-built technology.
> All your assumptions are predicated on the ability to emulate a human in software
Several of my assumptions are, true, though I can imagine other possibilities that would have similar effect.
> An even larger assumption is that it's even possible to do that.
That's not a larger assumption, that's the same assumption I stated.
> Can an intelligence program something smarter than itself? I don't think so.
On what basis?
> I don't think humans can.
Humans don't have to build a system smarter than ourselves, just build one as smart as ourselves and run it a lot faster with access to a lot more data.
Or in other words AI has nothing to do with nukes since what AI does has no bearing on if nukes will be set off.
Who's to say strong AI will even be significantly smarter than us anyway? What if humans are almost as smart (obviously there's some room for improvement) as it's physically possible to be? Truth be told we really don't know that much about consciousness or intelligence.
By definition of strong AI: AI capable of self-improvement past human levels; the assumption in my post is that it's possible to build strong AI.
> What if humans are almost as smart (obviously there's some room for improvement) as it's physically possible to be? Truth be told we really don't know that much about consciousness or intelligence.
We don't know that much, but we know enough to make reasonable predictions, and from all the available information, it seems rather exceedingly unlikely that humanity is the ultimate unsurpassable plateau of sapient thought. We have approximately zero evidence to support a hypothesis that humanity is anywhere near the physical limits on the capabilities of a mind, and plenty of evidence to support the hypothesis that we're not even close.
Wait, that doesn't make sense.
Interesting, I hadn't seen that one. While that doesn't give a very good impression of what the timescale could be (though it actually seems fairly plausible for an uploaded human that just runs faster with more data but doesn't attempt iterative self-improvement), it does suggest at least some possibilities in ways I haven't seen in other movies, and in relatively few short stories.
> That said, I don't think I'd constitute the conflation of any human mind and supercomputing as friendly.
"Friendly AI" is a specific term of art, referring to AI that values sentient beings, and whose value function involves modeling and satisfying the value functions of sentient beings. (Its opposite, "unfriendly AI", refers to any AI that doesn't value sentient beings; that doesn't just mean "actively hostile", but also simply "doesn't explicitly care", either one of which would create an existential threat. For instance, a paperclip-building AI that sees people as another source of matter that isn't paperclips yet.)
Almost all sentient beings tend to hold their own continued existence in high value.
Comparatively, teaching/programming a system to have morality seems like a safer bet than giving the reins to a being whose psyche was sculpted by evolution.
Because living batteries are clearly absurd.