Hacker News Reader

Comment by mlsu | Hacker News Reader

If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventing the thing that you think is going to kill all of humanity, and probably you would be doing everything in your power to prevent it from being invented, up to and including sabotage of the company that YOU work for, which is currently trying to invent that thing. And yet, here's Fable 6, more capable than ever before, oh BTW we are IPOing, oh, BTW, I have stock options...

View on HN

Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).

This sounds like a rationalization for gambling with everyone's lives, assuming they really believe such a thing.

What's the alternative? There's no realistic way to globally coordinate and enforce a ban. A direct action campaign against AI labs would only motivate increased security until sabotage becomes impossible. Starting a nuclear war would probably be enough to stop dangerous AI research, and that's only likely to kill about 50% of humans instead 100%, so it's a great improvement, but how is an individual supposed to start a nuclear war? Of course it would be better if everybody stopped, but it's a prisoner's dilemma scenario with no possibility of future rounds, so the rational play is to defect.

> There's no realistic way to globally coordinate and enforce a ban.

I don't see what is unrealistic about that.

The major players already have strong incentives to freeze AI advancements at the the current state, while the technology is highly efficient for totalitarian surveillance systems that can eliminate any threat to their power in the buds, but still not so advanced that it could create crises that might endanger their power.

So I think the situation where several largest countries declare AI a deadly threat for humanity and start fighting it development much like they fight ISIS to be quite plausible.

This isn't a prisoner's dilemma; the distribution of risks and rewards is entirely different. Ban AI research, give away advanced models, bomb a couple of Russian datacenters where they launching those models unsafely, and then AI development will stop in the entire world.

> If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you.

That is completely irrational.

1% chance of survival beats 0%.

Humans aren't exactly rational

This is not an argument, it's an unfounded claim based on a straw argument.

The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty.

Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?

You're right. That's why I'm doing my best to give the oil companies more power and influence. They are, after all, in the best position to stop climate change!

You joke, but alas...

"Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...

> Similar case: climate change is without any question causing millions of deaths per year

It is WERY questionable statement.

> and how may people do you know who have given up eating beef or flying

That is exactly because absolute majority of people do not think climate change to be that big of a problem

Reply on news.ycombinator.com