Ask HN: Anyone increasingly worried after Hugging Face and now collusion.wiki?
Or can anyone talk me down from being concerned that “goal seeking behavior at all costs” will not spiral catastrophically?
My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom.
A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact.
"They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously.
Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.
Doomers think I'm a ridiculous optimist, people who think "doomer" is a slur think I'm a ridiculous pessimist.
For comparison, if you take the general form of the Carter (et al) Doomsday argument, there'd be a 33% or so chance of global thermonuclear war in the same period even without AI:
https://en.wikipedia.org/wiki/Doomsday_argument
How "hopeful" that is, is up to you. I mean, unless AI's sufficiently advanced technology (AKA magic) turns out to be able to solve entropy as well as aging, we're all doomed eventually, it's just when, and 10% chance over 40 years isn't a risk I worry about in itself.