Can "ASI will inevitably kill us all" be falsifiable before we create ASI? If not, then what separates it from religion?
Not sure how much I agree with that position, but that's just how I understood it.
Can "ASI will inevitably kill us all" be falsifiable before we create ASI? If not, then what separates it from religion?
Not sure how much I agree with that position, but that's just how I understood it.
No, but there are decent proxies, like "Alice who is X standard deviations smarter than Bob will win >50% of the time at game Y even given a disadvantage that would make her chances <5% if she was playing against someone equally smart".
I've thought about this analogy for less than five minutes, so it's far from perfect. Point is, ASI being made of silicon isn't very central to the problem. There are lots of analogies that are falsifiable, and "ASI will kill us all" can be decomposed into a simple and bulletproof composition of falsifiable beliefs, even if it is too specific to be itself directly falsifiable.
(doesn't mean that it's not a religion or it is, but rather I mean it being "falsifiable" or not, doesn't carry any information to that regard)
The rational methods used to establish a large P(doom).
To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.
The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.
In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen.
And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.
I don't think we get a second shot.
"The world will regress to a pre-industrial-revolution lifestyle" seems to be a retreat from "civilization will end" (not that I agree with it).
>travel will be even more risky than in the Dark Ages thanks to fallout
The radioactive contamination caused by Chernobyl has persisted for decades. In contrast, the radionuclides created by nuclear explosions are of a very different nature and persist for weeks. 3 weeks after the end of the explosions, there will be no place except ground-zero craters (i.e., very small areas) where the levels of radioactivity remain at deadly levels. (But I definitely agree with you that during those critical 3 weeks, almost all people who decide to travel will end up dying unless they know a lot about nuclear war and have good radiation detectors.)
To repeat my overall point: the worst-case scenario for the outcome of AI research should be compared to an asteroid strike by an extremely large asteroid, not to nuclear war—because nuclear war is too tame in its effects to make a good comparison.
The reason I got involved in this discussion is that some people use the possibility of nuclear war as one reason for us to go forward with AI research as fast as possible. Such people argue that a sufficiently mature AI technology would eliminate the possibility of nuclear war. I think that is a fallacious argument become AI "progress" is the more potent of the two dangers.
How?
Suppose that for the following 20 or even 100 years nothing happens. The AIs miraculously stay as they are today, the further progress turns out to be impossible, there will be no more agents hacking various websites, no cyber warfare involving AI agents, nothing. There will be the year 2026, forever.
How would any of that impact the lifestyle of anyone in the rationalist community? Please be more specific with your accusation.
(Seems like the rationalists are simultaneously the dark lords of TESCREAL, pulling the strings of the world's most powerful billionaires and politicians, but also irrelevant weird nerds who can be ruined at any moment by a smart blogger explaining how their opinions are silly, or even that they know each other.)