Therefore, something will go wrong (invalid conclusion).
https://en.wikipedia.org/wiki/Appeal_to_probability
Sure seems to me that the whole Lesswrong crowd has forgotten their founding principles around cognitive biases. Or maybe they're just having a bit of fun riding the hypecycle.
The magnitude of the perceived risk matters.
You're defending an appeal to probability fallacy with a false equivalence fallacy.
The outcome of the thing going wrong is literally the worst thing that can happen to the entire humanity.
Therefore we should do everything in our power to avoid doing that thing untill we can eliminate the risk.
I acknowledge that it's a possibility; that's not the argument at hand.
There's also evidence that progress is exceeding people's expectations and we might not even realize what we have is dangerous untill too late.
If we had a probable collision trajectory of a meteor or signs of Yellowstone activating then yeah those would be good ideas.
"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing."
This is Ridley Scott, not Aristotle.
The same hyperbole was employed by Ehrlich in the late 60s when we were talking about the population bomb, or Nevil Shute in 1945 on the nuclear apocalypse - both scenarios seem to be increasingly unlikely.
I think we can both agree on the fallibility of human beings – but the leaps of logic require such a tenuous series of gymnastic leaps that the polemic is not only hard to take seriously, but indeed can cloud our ability to perceive actual threats!
From Eliezer's recent article in TIME:
"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...)
C'mon.
If you see fresh leopard tracks maybe it hasn't smelled you yet, but you shouldn't wait to start readying your rifle ready until it making a pounce at you.
I also would definitely not call the risk of nuclear war decreasing, with the Ukraine conflict ongoing and multiple nuclear powers experiencing growing instability. The number of friends I have who are knowledgeable about surviving a nuclear strike and/or own a CBRN mask went from 25% to 90% last year.
The illusion of control over something that may well have incomprehensible intelligence is a joke. Tell me, what control does the farm animal have over its own fate?
Obviously there is a wide range of possibilities from the utopian, to the benign, to the apocalyptic.